What's the best way to share data?

Selecting the appropriate data format for sharing is a pivotal decision influenced by the target audience and purpose. Formats like CSV, JSON, XML, or HDF5 suit raw or structured data, while HTML, JavaScript, or Jupyter notebooks are optimal for interactive or visual data. For reproducible or executable data, R Markdown, Python scripts, or Docker containers are apt choices. The key lies in choosing a compatible, accessible, and understandable format to effectively convey your data's insights to the intended recipients. Remember that the chosen format should facilitate seamless comprehension and utilization of the shared data by ensuring it aligns with the recipients' needs and technological capabilities.

From my experience, the choice of data format is often intertwined with the tools and infrastructure being used. Selecting widely-supported formats like CSV or JSON helps ensure accessibility, but sometimes efficiency or specialised needs dictate choices like HDF5 or Parquet. It's essential to provide accompanying documentation that clearly explains the structure and content, aiding those who are unfamiliar with the particular format you choose.

Here are the exact use cases of the given data formats- A) Raw or Structured Data 1. CSV: For structured tabular data 2. JSON: Excellent for compact, human-readable data exchange 3. XML: For complex, hierarchical data 4. HDF5: For Large, multidimensional scientific datasets B) Interactive or Visual Data 1. HTML/JavaScript: For web-based interactive data presentation 2. Jupyter notebooks: Combine code and explanations for data analysis C) Executable Data 1. Python scripts, R Markdown: Share data with analysis code 2. Docker Containers: Package complete environments for complete reproducibility

1. JSON (JavaScript Object Notation): A lightweight data interchange format widely used for transmitting data between a server and a web application, as well as for configuration files. 2. XML (eXtensible Markup Language): Another format for representing structured data, often used in web services, configuration files, and data storage. 3. CSV (Comma-Separated Values): A simple text-based format used for tabular data, where each row represents a record, and columns are separated by commas. 4. Excel (XLSX): Microsoft Excel's proprietary format for storing spreadsheet data. 5. HDF5 (Hierarchical Data Format version 5)

The subsequent data formats find extensive utilization in prevalent cloud services, serving as the foundation for data storage, sharing, and infrastructure utilization. They are, Parquet: This stands as a prominent columnar storage format, finely tuned for analytics and data processing. Its exceptional efficiency with sizable datasets makes it a frequent choice in data warehouses and Bigdata frameworks TSV: Tab-Separated Values(TSV) format is a type of data format commonly used for organizing and storing tabular data. In this format, each line of the file represents a record, and the values within a record are separated by tabs. YAML: YAML is a human-readable data serialization format often used for configuration files and data exchange.

APIs are the backbone of data democratisation and hence, the data-driven organization. The democratisation of data enables easy sharing of data with the business teams in a format they feel comfortable with. Given the native nature of APIs, they can be integrated with data products, ML models, and business intelligence dashboards. In addition, these APIs can be protected within data authentication and authorization layer. Data sharing in folders or central repository could a good start but quite tedious to maintain and operationalise as the volume grows.

Before we focus on the aspects of how we technically share data, it is critical to define the purpose of sharing the data and identify the specific datasets that are relevant to that purpose. Not all data needs to be shared, so focus on what is essential for the intended consumers. One of the critical aspects of data sharing, beside the format are data standards that ensure uniformity and consistency in the way data is formatted, labeled, and represented. This includes naming conventions, units of measurement, and date formats.

These are all great platforms, but the HuggingFace Hub has started to push ahead of the rest as the most accessible and straightforward platform to share datasets.

1. Data Integration: Data platforms can ingest data from various sources, including databases, APIs, streaming data, files, and cloud storage. 2. Data Storage: Data platforms offer scalable and reliable storage solutions for structured, semi-structured, and unstructured data. 3. Data Processing and ETL: Data platforms often provide tools for data transformation, cleaning, and enrichment. 4. Data Governance and Security: Data platforms ensure data security, compliance, and access control. 5. Data Analytics and Visualization: Data platforms enable data exploration, ad-hoc querying, and data visualization to gain insights from the data.

There are several cloud data platforms that facilitate data sharing and collaboration among teams and organizations. These platforms provides tools and services that enable secure and efficient data sharing, integration and analysis. AWS: 1.) Amazon S3 2.) Amazon Redshift 3.) AWS Data Exchange GCP: 1.) Google Cloud Storage 2.) BigQuery 3.) Google Cloud Data Transfer Service Microsoft Azure: 1.) Azure Data Lake Storage 2.) Azure SQL Data Warehouse 3.) Azure Data Share Snowflake: Snowflake's cloud based data platform offers a data sharing feature that allows organizations to securely share data. Databricks: Databricks also provides a collaborative platform for data engineering and analytics.

1. Public Domain: Data released into the public domain is not subject to copyright or any other intellectual property rights. 2. Creative Commons Licenses: Creative Commons licenses provide a range of permissions and restrictions depending on the specific license chosen by the data owner. 3. Open Data Commons Licenses: These licenses are designed specifically for open data and often include licenses like the Open Data Commons Public Domain Dedication and License (PDDL) and the Open Data Commons Attribution License (ODC-BY). 4. Proprietary Licenses: Some data owners may choose to keep their data under proprietary licenses, which restrict certain uses and may require payment or specific permissions for use.

The best practice is to have explicit data licenses - I'm moving from CC BY to more granular licenses like CC BY-NC-SA. For sensitive data, I work with legal to draft Data Usage Agreements to specify allowed uses.

Here is my practical advice on data licenses - 1. Be aware of the licensing conditions. Eg. Creative Commons licenses may call for acknowledgment. 2. Always read license terms. Some might prohibit derivative works or commercial use. 3. Give Proper Credit by including author's name, the type of license, and a link to the original data. 4. When integrating data, make sure the licenses are compatible. 5. Data Privacy: When handling sensitive data, such as patient records or consumer information, you must adhere to privacy laws. 6. Seek Legal Advice: If you have sophisticated data, you should seek legal help. 7. Keep up with any modifications to data licensing or conditions of use, especially if you use third-party data sources or APIs.

Public Domain (CC0): No copyright restrictions; data can be used freely by anyone. Attribution License (CC BY): Allows use, distribution, and modification, as long as original creators are credited. Attribution-ShareAlike (CC BY-SA): Similar to CC BY, but derivatives must be shared under the same license. Attribution-NoDerivs (CC BY-ND): Use and share freely, but no modifications allowed. Attribution-NonCommercial (CC BY-NC): Only non-commercial use allowed, with attribution to the creator. Open Database License (ODbL): Specific for databases; users are free to use, modify, and share as long as they attribute and share-alike. MIT License: Permissive and simple, often used for software and datasets, allowing almost unrestricted freedom.

There are several aspects that define data quality: 1. Accuracy: Data should be free from errors and represent the true values or facts of the entities or events it describes. 2. Completeness: Data should be comprehensive and include all the necessary information without significant gaps. 3. Consistency: Data should be consistent within itself and across different datasets. 4. Timeliness: Data should be up-to-date and relevant for the current context. 5. Relevance: Data should be directly related to the specific purpose or problem it is intended to address. 6. Validity: Data should adhere to predefined rules and constraints.

Great data tools like Pandas, Great Expectations, and Monte Carlo Data Profiling help automate data validation and quality checks at scale. I generate data quality reports for consumers on missing values, outliers, ID duplications, etc.

Proper documentation of the data (including metadata, data dictionary, and ETL details) helps the end-users understand the data more efficiently.

Here's some practical advice: 1. To get rid of errors and inconsistencies, spend money on data cleaning and validation methods. 2. Maintain and update your datasets frequently to ensure correctness. 3. To find and fix abnormalities, make use of tools and techniques like data profiling and outlier identification. For instance, while evaluating sales data, make sure there are no duplicate entries or missing numbers and that entries are consistent. 4. Always keep in mind that having access to high-quality data is necessary for making educated decisions.

Maintaining data quality is essential throughout the entire data management lifecycle, and not just when you're preparing to share it. Nevertheless, there are a few targeted data quality measures that should be put in place to identify any issues or inconsistencies introduced during the data sharing process: - Data Transformation Integrity - Data Mapping Verification - Consistency Across Systems - Temporal Consistency - Data Privacy Preservation Few other aspects to consider in the article would be: data lineage, and data observability that are critical for data quality management.

Consent and Privacy: Ensure data is collected and shared with informed consent, respecting individuals' privacy. Anonymization: Remove or obscure personal identifiers to protect individual privacy. Compliance with Laws: Adhere to relevant data protection laws like GDPR, HIPAA. Transparency: Be clear about data collection methods, purpose, and how the data will be used. Equity and Fairness: Avoid biases in data that could lead to unfair treatment of certain groups. Responsible Use: Ensure data is used ethically, avoiding misuse or harmful applications. Accessibility: Make data available to a diverse set of users, promoting inclusivity and broad utility. Ethical sharing balances the utility of data with the rights and dignity of individuals.

Here are some data ethics principles that I would like to delve into, to gain better understanding from individual's perspective. Privacy: Honoring individual's rights to manage their personal information and ensuring that data is gathered and employed solely for lawful and explicitly stated intentions. Transparency: Being open and honest about data collection practices, how data is used, and any potential implications for individuals. Consent: Securing well-informed and voluntary agreement from individuals prior to the collection and utilization of their data, while also granting them the freedom to revoke their consent whenever they wish.

Navigating the landscape of data ethics requires a deep understanding of both technical aspects and human-centric considerations. In my work, this has often meant putting in place strong privacy controls and thinking critically about potential biases or unfair impacts of data and algorithms. Engaging with a diverse team, including ethics specialists when available, has been crucial in building responsible data practices.

Beyond anonymization, I apply techniques like data blurring, noise injection, and federated learning to share useful data while preserving privacy. I conduct algorithm audits checking for bias and include ethical statements on sources and collection.

The moral compass of our data-driven society is data ethics. Here's some practical advise: 1. Transparency is the first piece of sound advice; be clear about how data is gathered, handled, and shared. 2. Give data protection high priority, especially when working with sensitive information. 3. Whenever necessary, obtain fully informed consent and anonymize data. 4. Make sure algorithms and decision-making procedures are fair; conduct regular audits. 5. Make sure, for instance, that recruiting practices don't bias against particular groups when using automated resume screening technologies. By upholding data ethics, we not only foster trust but also guarantee that the decisions we make using data are fair and just.

The other key aspects that drive effective story telling on data are, Interactivity:It's a best practice to provide interactive elements which allow audience to explore the data on their own terms. This will enhance engagement and understanding. Feedback: Always test your data story with small group and gather feedback. Iterate and adjust your feedback based on their input to present with more clarity and impact. Emotion and Connection: Use anecdotes, real-life examples and relatable scenarios to evoke emotion and connect with audience on personal level. Call to action: Depending on the goal, include a clear and concise call to action. This will enable the audience to take informed decisions based on the gained insights.

One thing I keep seeing with data storytelling is focusing solely on the data modalities, which is clearly wrong! People don’t care about how cool the technique you’re using. They do care about your story. How the story starts, how intruguing it is, how it evolves and concludes. These are main aspects of great data storytelling and storytelling in general. So, while building your stories try to spend time on “how” and “why” as much as “what”, maybe even more.

Best tool to make use of data for insights is Power BI or any reporting tool. It provides in depth insights to the data for the end user. It can also be automated using Machine learning algorithm for automation

📊 Unlocking the Power of Data Storytelling 📊 In today's data-driven world, there's no denying that data is the key to informed decision-making. But there's an often-overlooked aspect of this data revolution - storytelling. 📢 The Missing Piece: Data Storytelling Many data professionals lack the skill of storytelling and effective presentation. It's not just about using BI platforms; it's about presenting data in a way that engages. 🎯 A Call to Action We should collectively shift our focus towards training data professionals to be not just analysts but also owners and presenters of data, given they are the builders and experts of it. #DataStorytelling #DataAnalytics #DecisionMaking

A common theme across all these aspects is the need for a clear understanding of the audience and the context in which the data will be used. This guides decisions from formats to ethics. Practical application requires flexibility and adaptability to the ever-changing technological landscape, and collaboration across disciplines is often essential to success. Emphasising education and clear communication can break down barriers and lead to more effective data sharing.

It depends on the "sharing purpose". In the past I have managed a 3 years project where non-competitors companies had to share their own data to understand better their customers. Let's imagine a company selling home fornitures sharing data with a TV producer: they are not competitor but the info owned by one company could be relevant for the others to create better segmentations. To be GDPR compliant we have created a "statistical digital twin" of the entire population of our country (+60mln individuals), recreating families and personal info (income, studies, spending, etc.) using hyper-local open data. Then we have done a statistical match of every company's customer with our stereotypes to have enriched and sharable profiles.

One technology I have found useful to share data securely is blockchain with smart contracts. By leveraging the inherent security and immutability of blockchain, smart contracts enable transparent and tamper-proof agreements that automatically execute when predefined conditions are met. This innovation ensures that parties involved can securely share sensitive data, confident that the information will be accurately recorded and accessible only to authorized participants. The decentralized nature of blockchain eliminates the need for intermediaries, reducing delays and costs associated with traditional data sharing methods.

It depends on the platform to share the data. If it's a public platform Kaggle comes to mind. Data World could be other option. LinkedIn obviously. For everyone to reach the data, social media platforms could be an alternative to all those places. Also, the size of the data enters this conversation as well. Kaggle one of the only places that handles such big data. In addition, Data World presents huge amount of space for sharing the data. In terms of processing the data, again, Kaggle gives its users to do their work. I realized that I never mentioned the GitHub which is useful place. Data Protection laws deserve final place in this topic. To person who will share the data must read at least 5-6 different data protection laws.

Consider adding more on: data security (incl. compliance with GDPR, HIPAA, or CCPA), data access and retention policies, which all seem to be slightly underrepresented in the article.