data subTLDR week 42 year 2025

Many Reddit users expressed frustration about working on projects or tasks that ultimately went unused or unnoticed, including dashboards and software. These sentiments were often tied to feelings of underappreciation and wasted effort. One user highlighted the issue of imposter syndrome, indicating the disconnect between effort, results, and personal perception of achievement. Another user's anecdote underscored the potential negative impact of poor communication, leading to wasted time and resources. The overall tone was negative, reflecting dissatisfaction with certain workplace dynamics and the lack of recognition for effort and performance.

There's significant criticism around Fivetran's open data infrastructure, with many expressing dissatisfaction and frustration. Users feel that the term open is being misused and that companies are prioritizing profits over the spirit of open-source software. Some comments suggest that developers are selling out, allowing companies to manipulate their projects and market demand. However, a counterpoint highlights that Fivetran can't revoke rights to projects already released under open-source licenses, and that the main concern should be potential lack of future development or support. Overall, the sentiment is overwhelmingly negative, reflecting deep concerns about the future of open-source in the data engineering space.

The Fivetran and DBT merger is seen as a strategic move to challenge the dominance of data warehouses such as Databricks and Snowflake. The new entity, DBTran, aims to commoditize these warehouses by integrating different tools and technologies. However, opinions are mixed, with some users suggesting Fivetran's survival and competitive edge are at stake. Concerns were raised about Fivetran's ability to outperform Snowflake or Databricks in cataloging and platform capacities. Some foresee another merger with either Databricks or Snowflake in the next few years. The overall sentiment is cautious, with an undertone of skepticism about the success of the merger.

The popularity of Monte Carlo (MC) methods over Polynomial Chaos Expansion (PCE) in solving stochastic problems stems from MC's simplicity, flexibility, and ease of application across a wide range of problems, even in high dimensions. While PCE can offer greater efficiency and accuracy, its implementation is more complex, requires more upfront math work, and may not scale well to complex or high-dimensional systems. Users also noted that MC handles multimodality and non-smooth functions better, while PCE can be slow with many samples. Some, however, expressed curiosity and interest in exploring PCE further, suggesting a potential for increased use in the future.

Despite disappointment over a rejected application for an Applied Scientist role at Microsoft, the individual received encouragement to persist with applications as job acquisition is largely a numbers game. It was noted that Applied Scientist roles usually require a PhD or equivalent experience. The individual was also encouraged to expand their skill set by participating in open research projects and consider applying for Applied AI Engineer roles, which may better suit their current experience. A shift in career direction may call for adjustments in the current role or a willingness to make lateral or upward shifts.

The discussion reflects a positive sentiment towards SQL proficiency as a result of regular use and problem-solving rather than formal expertise. Many participants shared their experiences of still needing to refer to resources for certain functions irrespective of their years of experience, suggesting that SQL has a steep and continuous learning curve. Participants also emphasized the importance of context knowledge in SQL, noting that understanding the data and requirements often poses greater challenges than the SQL commands themselves. The consensus was that SQL proficiency comes with time and regular use, and it's not unusual to continue learning and consulting resources, even with experience.

In a discussion about using outside resources during coding tests, most participants agree that understanding the concept is more important than memorizing the syntax. They often look up syntax in their daily work, and assert that the ability to solve problems in a real job doesn't hinge on rote memory. Some express disdain for live-coding interviews under scrutiny. A few also warn against over-reliance on AI or friends, emphasizing the importance of understanding programming language to work effectively with data. Overall, the sentiment is positive towards using resources to recall syntax during a coding test, as it mirrors real work situations.

Learning SQL is best achieved through repetitive practice and practical application. Download SQL server developer edition and sample databases, such as AdventureWorks DB or StackOverflow's public database, to gain hands-on experience. Set up tables and practice CRUD operations while studying. Having a project or goal aids in learning syntax and concepts. Use public data and create simple tables/graphs for better understanding. Utilizing datasets related to your hobbies can make the learning process more engaging. Understanding SQL as a language for selecting subsets from tables, rather than procedural programming, can simplify the process. Overall, a creative application of knowledge through curiosity about each command's function and limitations aids retention.