data subTLDR week 28 year 2026

Concerns are raised about the proposed replacement of BI tools with Claude. Many respondents advocate for better version control, with humorous examples of convoluted file names highlighting the issue. One comment suggests a preference for version numbers, while another emphasizes the need for proper version control. There's also a sense of frustration around changing requirements from different PowerBI users who each consider themselves the core user. Meanwhile, a few individuals seem either unfamiliar with or indifferent to spreadsheet usage. The overall sentiment leans towards skepticism and frustration regarding the proposed change.

The sentiment in the r/dataengineering thread is mixed; while some find enjoyment in the shared meme, others use it as a launching pad for serious discussion. The most upvoted comment emphasizes that the goal of data engineering isn't mastering complex mathematical concepts but rather ensuring the satisfaction of business users. The comment suggests that while creating efficient pipelines is important, it's often secondary to user happiness. Some comments also touch on the use of visual pipeline tools and the challenges of certain tasks. A few even express self-doubt, questioning their career choices and the tasks they are asked to perform.

The discussion centers on skepticism and criticism towards a prominent AI company's need for data engineers. Many commenters express concern over the short job duration and overly rigorous screening for a junior role, suggesting that the company undervalues data engineering work and may have financial constraints. There's a general sentiment that the company's attitude is arrogant and off-putting. A few commenters appreciate the company's honesty, and one suggests that the future may involve fewer data engineers, not none, based on their experience. Overall, the thread reflects a negative sentiment towards the company's approach.

The lack of consistent definition for the title 'Data Engineer' (DE) is causing confusion among job applicants and employers, leading to mismatches in skills and expectations. Some suggest that the issue lies not with the applicants but with the ambiguous use of titles in the industry. While the path to DE often progresses from data analyst to data scientist, the skillset required is closer to software engineering, making the jump from analyst to DE challenging. There is also a belief that software engineers transitioning to DE roles tend to be stronger candidates. However, the value of diverse skills in the market is highlighted, and companies are advised to clearly define job requirements and not devalue the work of other professionals in the industry.

The machine learning industry is facing criticism for its increasingly demanding job requirements, with the qualifications for many roles appearing nearly impossible to meet. Some professionals argue that these requirements, which often include deep expertise in multiple highly specialized fields, are unrealistic. Critics liken it to asking for a jack of all trades in highly specialized academic domains. However, others argue that the terminology is often misunderstood, with phrases such as deep expertise merely indicating significant work experience, and job descriptions often listing most rather than all necessary skills. The discussion suggests a gap between HR language and industry understanding, as well as the need for realistic expectations in recruitment.

The discovery of O’Reilly books on Machine Learning (ML) in a public library sparked a positive discussion. A substantial number of people highlighted the increasing accessibility of high-quality tech literature at public libraries. Many lauded the role of libraries in democratizing knowledge and bridging the digital divide. Some shared personal experiences of using libraries for self-education in tech fields. However, a few voiced concerns about outdated information due to the fast-paced nature of the tech industry. Overall, the sentiment was positive, reinforcing the value of libraries in providing quality, accessible resources for learning.

The machine learning community is showing interest in TorchJD, a recently improved library for training models with multiple losses in PyTorch. It offers both scalarization and Jacobian descent methods, with the latter being praised for its memory efficiency and speed, even reducing memory usage by up to 93% in some cases. Some users voice concerns about the trade-off between memory usage, convergence speed, and batch sizes, suggesting a need for more empirical results on multi-objective optimization rather than convergence speed. The creators acknowledge these concerns and encourage users to experiment with the library to assess its benefits for their specific needs. Overall, the sentiment is positive, with users appreciating the library's accessibility and potential for optimizing model training.

The thread revolves around the efficacy of the EXISTS clause in SQL over LEFT JOIN to avoid duplicate rows in complex join conditions. Users highlight EXISTS as a more efficient operation, as it stops after finding one matching value. It's noted as the best tool for existence checks without needing to return or use any columns from the second table. EXISTS is also useful in automation and processes like WHILE EXISTS loops, and is more efficient with indexed fields. However, if a column from the secondary table is needed or to continue joining to a third table, a join is recommended. Users also suggest looking into semi-join and anti-join instead of focusing on specific keywords.

The thread highlighted the importance of regularly checking pg_stat_statements in databases, as the most significant database compute drain was discovered through it rather than slow-page symptoms. A non-indexed UPDATE query was found to be quietly consuming significant compute resources, despite not appearing slow or generating errors. The consensus was that queries should be ranked by total execution time, not just by visible speed. Adding indexes was strongly recommended, particularly with 'CREATE INDEX CONCURRENTLY' for large tables, to avoid outages. The overall sentiment was mixed, emphasizing the need for vigilance and correct indexing in databases.

There's a mixed sentiment towards a take-home SQL interview for data analysts. Some users are surprised that SQL is still used in an era dominated by AI, while others find such interviews more comfortable than live interactions. The importance of analyzing and interpreting data, not just writing SQL, is highly emphasized. Users note that valuable skills include spotting irregularities like date formatting issues, missing values, and explaining how they are handled. Additionally, the ability to use AI to scrutinize results and analyze SQL outputs is considered crucial. The expectation for candidates to explain their SQL queries and results is seen as a key part of the process.