data subTLDR week 38 year 2026

The TMLR journal attempted to engage authors of 10 papers facing desk rejection, revealing concerning results. Many authors struggled to explain their papers, either due to unavailability, inability to answer basic questions, or difficulty discussing technical details. One author could answer all questions but had a major flaw in their work. The majority of comments highlighted a need for change in academic peer review and suggested that even accepted papers might show similar issues. Some users supported a more rigorous review process, but others criticized TMLR's approach as discouraging or inefficient. The sentiment was mixed, reflecting broad concerns about academic integrity and review practices.

The discussion revolves around a research paper suggesting that recursive self-improvement (RSI) in AI is not imminent as current AI models fail to conduct open-ended machine learning (ML) research. The sentiment is mixed, with some commenters questioning the paper's methodology and conclusions, while others agree that current models are limited in their capabilities. Criticism includes the notion that AI's role in accelerating research is already substantial, and that not being able to self-improve now doesn't mean it's not an upcoming possibility. Supporters highlight that AI's capacity for RSI requires continuous data addition, environmental feedback, and computational power, adding that this won't spontaneously happen with large language models (LLMs). The discussion also touched on the potential of transformer architectures in AI research.

Many Reddit users have shared shocking workplace experiences in the tech industry, with the most upvoted stories revealing concerning practices in data management. One user cited a failed company that bizarrely used Kafka as a Postgres database and stored all table data in a single JSON column. Other users pointed out inefficient practices like performing all transformations in Power BI, causing projects to take hours to open. Another user discovered a table for credit card charges that was updated/deleted instead of using an append-only approach, raising suspicions of potential embezzlement. These anecdotes reveal an urgent need for better data management practices in the tech industry.

The discussion on data engineering in Microsoft Fabric reveals a mix of views. A widely supported recommendation is to use VS Code for data engineering, deploying to workspaces using `fabric-cicd`, and keeping the implementation code-first. Users suggest leveraging Fabric's automation tools, arguing that not doing so could waste the platform's value while not improving individual skills. There are also warnings against letting users create their own dataflows and caution around capacity usage and preview features. Some users have had positive experiences using Fabric with Azure Databricks or integrating SQL models. A minority find working with Fabric via VS Code challenging. Overall, the sentiment is mixed with a slight lean towards positive.

The SQL concept that took users the longest to truly understand, as revealed by upvotes, was the idea of joins. Many struggled with the different types of joins, such as inner, left, right, and full outer. The concept of indexing was another challenging area, with users highlighting its importance for performance optimization. Additionally, recursive queries and the application of group by and having clauses were found to be complex. The overall sentiment was mixed, showing a shared struggle in understanding these concepts but also a sense of achievement once mastered.

Data governance professionals on Reddit recommend creating a detailed data catalog, aligning data definitions across the organization, setting up data quality metrics, and ensuring clear communication about the purpose and fit of source data. Some mention the challenge of proving their value to the organization and the criticality of their role in preventing costly errors. They also emphasize their role in absorbing challenges to allow developers and engineers to work effectively. Overall, the sentiment is mixed, suggesting data governance is a demanding role that requires a balance of technical and business knowledge, strategic planning, and communication skills.

The key SQL query optimization technique valued by professionals is the ability to read execution plans, which allows users to determine where to apply an index, modify a query, or identify peak efficiency. Other techniques include the use of artificial intelligence to generate queries based on pre-existing knowledge bases, and shifting from common table expressions to temporary tables for improved performance. The use of views for complex data sets and staging data in 'waterfall' datasets are also noted as beneficial strategies. The sentiment is that understanding execution plans is essential for performance, and any other action is less effective without this knowledge.

In a discussion about the use of Common Table Expressions (CTEs), subqueries, and temporary tables in SQL, the sentiment leaned towards CTEs as a preferred method due to readability, optimization, and troubleshooting ease. However, some users argued that temporary tables provide better debug capabilities and can be more efficient for large queries or frequent data calls. Subqueries were mentioned as being used sparingly. The consensus was that each method has its place and the choice often depends on the specific situation and user's preference. Clean, logical SQL writing was advised over focusing too much on the choice between these methods.