
Flat Data eschews the complexity of many tools in the data/ETL space in favor of something simple and flexible enough for many workloads, but which requires no user-maintained infrastructure. While Flat has a ton of utility for developers, we want to make it easier for scientists, journalists, and other developer-adjacent audiences to develop lightweight, data-driven apps.

The Flat Data project incorporates three different pieces:

<div className="mt-6 mb-12 wide">
  <div className="grid md:grid-cols-3 gap-8">
    <ProjectLinkLargePill
      link="https://github.com/marketplace/actions/flat-data" title="Flat Action"
      color="indigo"
      subtitle="GitHub Action"
      description="The GitHub action which fetches and transforms data."
    />
    <ProjectLinkLargePill
      link="https://marketplace.visualstudio.com/items?itemName=GitHubOCTO.flat" title="Flat Editor"
      color="pink"
      subtitle="VSCode&nbsp;extension GitHub&nbsp;Codespaces"
      description="A graphical interface for authoring Flat Data workflows."
    />
    <ProjectLinkLargePill
      link="http://flatgithub.com/" title="Flat Viewer"
      color="yellow"
      subtitle="Website"
      description="Browse, filter, and share data from flatfiles in your repo."
    />
  </div>
</div>

<Callout icon="LawIcon">
  <p>Flat is an experiment from the Developer Experience team in GitHub's Office of the CTO and is not an official GitHub product. We are publishing it (as well as all documentation and examples) under the terms of the <a href="https://github.com/githubocto/flat/blob/main/LICENSE" target="_blank" rel="noopener noreferrer">MIT license</a>.</p>
</Callout>

## How to use Flat Data

Currently, Flat knows how to fetch data from two kinds of sources: any HTTP endpoint, or a SQL query against a [supported datastore](https://github.com/typeorm/typeorm/blob/master/src/driver/types/DatabaseType.ts). Let's start with the simpler of the two, an HTTP fetch!

<Aside>
  We jump right into <i>how</i> to use Flat Data, but feel free to <a href="#why-flat-data">click to skip down to <i>why</i> we built Flat Data</a>.
</Aside>

To get started with Flat Data, all you need is an endpoint or file that you want to download into a GitHub repository. In this case, we’ll be fetching the current Bitcoin price from <a href="http://api.coindesk.com/v2/bpi/currentprice.json" target="_blank" rel="noopener noreferrer">this link</a> every five minutes and storing that data in a GitHub repository. The complete example with a detailed tutorial can be found <a href="https://github.com/githubocto/flat-demo-bitcoin-price" target="_blank" rel="noopener noreferrer">here</a>.

### Part I: Get data into your GitHub repository

<div className="full">
  <ScrollyList
    doShowNumbers
    rightContent={index => (
      <div className="aspect-w-4 aspect-h-3 pt-2" key={index}>
        <video playsInline loop autoPlay muted controls>
          <source
            src={
              [
                '/assets/projects/flat/step1.webm',
                '/assets/projects/flat/step2.webm',
                '/assets/projects/flat/step3.webm',
              ][index]
            }
            type="video/webm"
          ></source>
          <source
            src={
              [
                '/assets/projects/flat/step1.mp4',
                '/assets/projects/flat/step2.mp4',
                '/assets/projects/flat/step3.mp4',
              ][index]
            }
            type="video/mp4"
          ></source>
        </video>
      </div>
    )}
    items={[
      <div>
        <div className="font-bold mb-2">
          Setting up a Flat Data GitHub Action
        </div>
        <div>
          The first thing we have to do is to specify what type of data we want
          to download into our GitHub repository. We add a new endpoint using
          the <b>+HTTP</b> button and fill in the relevant details: an
          endpoint's url and a name for the file we want to save.<br></br>
          <br></br>By default, our data will be updated every five minutes, but you
          can specify a different schedule or write a custom one.
        </div>
      </div>,
      <div>
        <div className="font-bold mb-2">Sync with GitHub</div>
        <div>
          Our GitHub Action is all set. We can upload the changes to our repo
          and it will automatically run the action for the first time. Take a
          look at the Action's output by clicking on the
          <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/runs/2451713859?check_suite_focus=true#step:4:57"
            target="_blank"
            rel="noopener noreferrer"
          >
            {' '}
            Actions tab{' '}
          </a>
          on our repo.
        </div>
      </div>,
      <div>
        <div className="font-bold mb-2">Check the downloaded data!</div>
        <div>
          Our repo now
          <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/blob/main/btc-price.json"
            target="_blank"
            rel="noopener noreferrer"
          >
            {' '}
            has a file{' '}
          </a>
          called <code>btc-price.json</code>
          with a copy of the data in the endpoint! Now that it's set up
          <em> you'll always have a fresh copy of the data</em>, since this
          Action will run every 5 minutes.
        </div>
      </div>,
    ]}
  />
</div>

We hope you’re already starting to see how the Flat Action can be useful. It allows you to download any kind of data (JSON, CSV, images, text files, zip folders, etc.) into your repositories at a repeatable schedule. As long as what you’re fetching has a URL, it can be downloaded into a GitHub repo.

<Callout icon="RocketIcon">

At this point you’ve used Flat Data to get data on a schedule and keep it updated into your repo. Congrats!

But Flat Data can do a bit more. Read the optional parts II and III to see how you can add an extra step to process and view your data in more advanced ways.
</Callout>

### Part II: Make the data your own <span className="ml-4 inline-block text-xs uppercase tracking-widest p-2 px-3 bg-gray-100 text-gray-800 rounded-sm">Optional</span>

What if you want to process or change the data in some way before it gets added to your repository? We’ve built in a way for you to run these types of tasks using the `postprocess` parameter in the Action. Let’s take a look at how to do that.

<div className="full">
  <ScrollyList
    doShowNumbers
    rightContent={index => (
      <div className="aspect-w-4 aspect-h-3 pt-2" key={index}>
        <video playsInline loop autoPlay muted controls>
          <source
            src={
              ['/assets/projects/flat/step4.webm', '/assets/projects/flat/step5.webm'][
                index
              ]
            }
            type="video/webm"
          ></source>
          <source
            src={
              ['/assets/projects/flat/step4.mp4', '/assets/projects/flat/step5.mp4'][
                index
              ]
            }
            type="video/mp4"
          ></source>
        </video>
      </div>
    )}
    items={[
      <div>
        <div className="font-bold mb-2">
          Create a postprocessing script to manipulate the data
        </div>
        <div>
          Let's say we only want a snippet of the data in our{' '}
          <a
            href="http://api.coindesk.com/v2/bpi/currentprice.json"
            target="_blank"
            rel="noopener noreferrer"
          >
            original endpoint
          </a>
          . We can create a <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/blob/main/postprocess.js"
            target="_blank"
            rel="noopener noreferrer"
          >
            <code>postprocessing.js</code>
          </a> or <code>postprocessing.ts</code>
          file written in{' '}
          <a
            href="https://deno.land/"
            target="_blank"
            rel="noopener noreferrer">
            <b>Deno</b>
          </a>{' '}
          to filter and save just the data we care about.<br></br>
          <br></br>Then we simply specify a path to the file in the
          <code>postprocess</code> parameter of the Flat Data Action.
        </div>
      </div>,
      <div>
        <div className="font-bold mb-2">
          Sync to GitHub to create a new filtered data file
        </div>
        <div>
          Let's{' '}
          <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/runs/2452396626?check_suite_focus=true#step:4:55"
            target="_blank"
            rel="noopener noreferrer"
          >
            re-run our Action
          </a>{' '}
          using our new{' '}
          <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/blob/main/postprocess.js"
            target="_blank"
            rel="noopener noreferrer"
          >
            postprocessing script
          </a>
          . This will run automatically, once the changes are committed & pushed
          to the repo. Now you'll see a new <a
            href="https://github.com/githubocto/flat-demo-bitcoin-price/blob/main/btc-price-postprocessed.json"
            target="_blank"
            rel="noopener noreferrer"
          >
            <code>btc-price-postprocessed.json</code>
          </a> file with our custom filtered data!
        </div>
      </div>,
    ]}
  />
</div>

### Part III: Visualizing our data for easy sharing <span className="ml-4 inline-block text-xs uppercase tracking-widest p-2 px-3 bg-gray-100 text-gray-800 rounded-sm">Optional</span>

Now that we have our data, what can we do with it? There are tons of options (throw it in a database, create a dashboard that uses it), but we just want to visualize and share it in a simple way.

For that we built a simple tool that takes any public GitHub repo and returns a nice GUI for viewing data. To use it, just prepend <code>flat</code> to any GitHub repo's URL. For example, <a href="https://flatgithub.com/the-pudding/data?filename=boybands%2Fboys.csv&sha=9b4e89ed178e11bf70a9b3d8ffa57ac2ea181b96" target="_blank" rel="noopener noreferrer">`https://flatgithub.com/the-pudding/data`</a>.

<div className="wide my-6">
  <iframe
    src="https://flatgithub.com/the-pudding/data?filename=boybands%2Fboys.csv&sha=9b4e89ed178e11bf70a9b3d8ffa57ac2ea181b96"
    width="100%"
    className="wide max-h-[35rem]"
    style={{ height: '90vh' }}
  />
</div>

You can use Flat Viewer to take a closer look at any tabular, non-nested CSV and JSON file that lives in a GitHub repo, like [these ones](https://github.com/the-pudding/data) from [The Pudding](https://pudding.cool/).

Once your Action has run a few times, you can see how the data has changed over time by clicking through the **Commits** dropdown. The table will highlight new, modified, and deleted lines for each commit.

## Examples

As we saw in our walkthrough, the basics are very simple. You can get a Flat Action up and running in a few minutes. But Flat Data can be very powerful!

Here are a few examples for your reference, and to show the breadth of what Flat Data can help you with. Even more examples can be found <a href="https://github.com/githubocto?q=flat-demo&type=&language=&sort=" target="_blank" rel="noopener noreferrer">here</a>.

<div className="wide mb-24">
  <FlatDataExamples />
</div>

## Why Flat Data?

There’s a certain kind of language that dominates the discourse on data today. Big data, bigger data, biggest data. A million rows aren't cool. You know what's cool? A _billion_ rows. Distributed data systems that slip the surly bonds of any one machine to touch the face of scale. Techniques for sampling and transforming data while it moves instead of operating on it after it has come to rest. Strategies for contending with a deluge of events from chatty devices. Directing those data tributaries into undifferentiated data lakes so that we may pose different queries to the data someday. Beyond the pain of actually doing this work, it has become impossible to talk about this stuff without abusing metaphors beyond their safe design limits.

As a developer, all those bits occupying the proverbial lake/warehouse/refinery are as immediately useful as a grape seed is to a winery. Locality of data isn't some abstract concept when you're trying to build things on top of that data — it's the leading term of developer experiences. If I have the data, I can load it, and get to work. If I don't have the data, then it doesn't matter if the data is cleaned, filtered, and sorted. If I don't have the data, then either I alter my application logic to work with data at a distance, or I figure out how to bring a working set to my local environment.

As a result, there's an entire industry of tools that accomplish the chore of getting data to the right place, in the right format, at the right time. Data architectures vary wildly, so these solutions have a wide range of ambition (what they attempt to do) and complexity (what mess they attempt to conceal.) This is in contrast to the application/compute space, where pithy, prescriptive manifestos like [12 factor](https://12factor.net) offer a great lens through which to think: "Do it this way, and thou shalt scale." There is no equivalent for data because there are so many different approaches; one can only nail so many theses to the church doors before running out of room to describe best practices.

It's easy to get dazzled by the complexity and diversity of data tooling, but we're not the first profession to invent hyper-specialized implements. Surgeons have trays full of weird tweezer-like things whose shapes differ but whose ultimate purpose is the same: grabbing hold of stuff that might be difficult to grab with our fingers. Surgery on data also requires specialized tools that offer unusual capabilities or guarantee certain behaviors! However, most medicine is not surgery, and we do ourselves a disservice if we accept that sort of complexity into every situation involving data.

Flat Data aims to simplify everyday data acquisition and cleanup tasks. It runs on GitHub Actions, so there's no infrastructure to provision and monitor. Each Flat workflow fetches the data you specify, and optionally executes a postprocessing script on the fetched data. The resulting data is committed to your repository if the new data is different, with a commit message summarizing the changes. Flat workflows usually run on a periodic timer, but can be triggered by a variety of stimuli, like changes to your code, or manual triggers. That's it! No complicated job dependency graphs or orchestrators. No dependencies, libraries, or package managers. No new mental model to learn and incorporate. Just evergreen data, right in your repo.

<DataFlowDiagram
inputs={['data source', 'data source', 'data source', 'data source']}
generated={['data.json,process.js,processed data.json']}
outputs={['Flat Viewer', 'Your app', 'Your database']}
/>

Flat stands squarely on the shoulders of the [git scraping pattern first articulated by Simon Willison](https://simonwillison.net/2020/Oct/9/git-scraping/). It makes this pattern dead simple for developers and accessible to other audiences that are already using GitHub to share and work with data, like journalists and scientists. With [codespaces](https://github.com/features/codespaces), you can author Flat data workflows and develop applications that use that data without ever leaving your browser, and without that data ever reaching your machine. If you have a static webapp, you can power it with data that is kept evergreen by Flat (we built a simple example using GitHub pages). You can use Flat to monitor systems in an auditable fashion, using git commits to track changes. Flat works for both textual or binary data, and is great for keeping test fixtures up to date.

<aside><a href="https://github.com/alex/nyt-2020-election-scraper">Alex Gaynor’s 2020 Election Tracker</a> was an immensely popular instance of this pattern, offering a realtime interface on top of the New York Times' data during the 2020 elections in the United&nbsp;States.</aside>

To abuse [the first law of thermodynamics](https://en.wikipedia.org/wiki/First_law_of_thermodynamics), you can't eliminate the need to fetch data in apps, but you _can_ shift that responsibility around. Instead of assembling data at runtime, we try to think of the ideal dataset that we want to use, and let Flat materialize that dataset for us.

We're excited to see what you do with it!

## Feedback

We'd love to hear how you're using (or would like to use) Flat Data! Tweet us at [@githubOCTO](https://twitter.com/githubOCTO) or start a thread in [Flat Data discussions](https://github.com/githubocto/flat/discussions).

Right now, you can configure Flat to fetch data from either a HTTP endpoint or SQL database, but we would love help extending that list! Click that fork button, add support for a different backend, and send us a PR!

✌️ ❤️

GitHub OCTO<br/>
Developer Experience Team

## From the Community


Flat explores how to make it easy to work with data in git and GitHub. It builds on the “[git scraping” approach pioneered by Simon Willison](https://simonwillison.net/2020/Oct/9/git-scraping/) to offer a simple pattern for bringing working datasets into your repositories and versioning them, because developing against local datasets is faster and easier than working with data over the wire.

Flat explores how to make it easy to work with data in git and GitHub, offering a simple pattern for bringing working datasets into your repositories and versioning them.

Flat Data


The rising complexity of modern software and data is pushing the limits of human understanding. While open source collaboration is connecting millions of developers and collaborators, we need new ways to explore data and make better decisions. Data visualization is changing how we understand and sharing insights about the world.

<hr/>

Melody Meckfessel is the CEO and co-founder of [Observable](https://observablehq.com), the destination for data visualization. Observable empowers people to explore, analyze and visualize data on the web, together. Melody is a hands-on leader with decades in engineering at both startups and enterprises. Prior to Observable, she was a VP of Engineering at Google.


Melody Meckfessel

CEO / Co-founder

How Visualization Is Changing How We Work With Data


When I think about open source sustainability, I think about money, sure. But I also think about what configurations of funding would be more likely to keep legacy infrastructure ticking along AND provide R&D opportunities for innovators; what tooling we need; how a stronger ecology of consultancies would change the interactions among volunteers, companies, and other institutions; etc. I'll discuss what I've learned about healthy maintainership, and what a healthier future would look like for the open source industry.

<hr/>

Sumana Harihareswara is a project manager and programmer who has worked on PyPI, Zulip, Mailman, MediaWiki, HTTPS Everywhere, and several other open source projects. Most recently she has managed improvements to pip's dependency resolver and user experience, expedited the release of new versions of pipenv and autoconf, and written a somewhat frightening number of successful grant proposals to fund urgent work in critical open source projects. She earned an Open Source Citizen Award in 2011 and a Google Open Source Peer Bonus in 2018. She lives in New York City and founded Changeset Consulting in 2015.


Sumana Harihareswara

Founder

What Would Open Source Look Like If It Were Healthy?


Who builds developer communities, why are companies investing in them, and why now?

<hr/>

swyx is passionate about Developer Tooling and Developer Communities. He is currently the head of developer experience at [temporal.io](https://temporal.io), and recently published the Coding Career Handbook for Junior to Senior developer careers. In his free time he teaches React, TypeScript, Storybook and Node.js CLI's at Egghead.io, and helps run the Svelte Society community of meetups.


swyx

Head of Developer Experience

Why Invest in Developer Community?


We were promised "bicycles for the mind", but the vision of computers as tools for thought remains elusive. In our pursuit of this ideal, we have identified design principles we call "local-first software" that can help our software run faster, last longer, and cost less to build. We'll start by talking about the ideas that motivate our research, then look at some real-world prototypes and the technologies behind them, and finally talk about the challenges still outstanding.

<hr/>

Peter van Hardenberg has worked as an Arctic oceanographer, a game developer, wrangled a (literal) million Postgres databases, and is now Executive Director of [Ink & Switch](https://inkandswitch.com) where he plots the future of tools for thought.


Peter van Hardenberg

Executive Director

Local-first Computing


What's next for JavaScript, web development, and the internet as a whole? Explore the global trends and specific technologies that will come to define the best tech stacks of this decade.

<hr/>

Fred K. Schott is the founder of [Skypack](https://skypack.dev/), a next-gen CDN for JavaScript and modern web development. Fred also created [Snowpack](https://snowpack.dev/), a modern build tool for web developers that leverages new JavaScript features to speed up traditional development workflows by 10x or more. Fred lives with his wife in Oakland, CA. He has spent the last 8 years working in JavaScript and Open Source Software at companies like Box, Google, and Ripple.


Fred K. Schott

FOSS Developer & Founder

JavaScript: The Next Ten Years


How do you build a solid DevRel organization? How do you measure and connect your DevRel org activities to the core outcomes of your company? Come learn form Amir Shevat, VP of Developer Experience at Twitch how to build, measure and justify an awesome Developer Relations org.

<hr/>

Amir Shevat is the co-founder and Chief Product Officer at [Reshuffle.com](https://reshuffle.com). Previously, Amir was the VP of Product at Amazon’s Twitch, Director of Developer Relations at Slack, Head of Scalable Developer Programs at Google, and Developer Advocate at Microsoft. Amir lives in Palo Alto, with his wife and two kids. In his spare time, he loves to bike and cook.


Amir Shevat

Cofounder

Building and Measuring Developer Relations

Every nerd deserves their own personal data warehouse - a system that gives them the same kind of analytical capability that is usually reserved for giant tech companies. I've figured out how to build one, using SQLite, Datasette and a handful of open source scripts and APIs.



I'll demonstrate my own personal Dogsheep, built using data from GitHub, Twitter, Foursquare, Apple Photos, Google, LinkedIn, 23AndMe and more. I'll show how legislation such as Europe's GDPR have made it easier for us to build our own personal analytics platforms, and discuss the way forward towards a world where we can empower even non-nerds to finally take control of their personal data.

<hr/>

Simon is the creator of Datasette, an open source tool for exploring and publishing data.

Datasette is based on Simon's experiences working as a data journalist at the UK's Guardian newspaper.

Simon is also a co-creator of the Django web framework. He recently completed the JSK Fellowship program at Stanford.

Simon Willison

FOSS Developer & Consultant

Personal Data Warehouses: Reclaiming Your Data

The UK is in the throes of a wholesale reform of school (K-12) education in computing, one that establishes computer science as a foundational discipline that all children learn from primary school onwards, just as they do maths and natural science. It’s pretty exciting: I will sketch the context and describe our progress.

But I will focus on one particularly exciting development: Project Quantum. So-called “diagnostic questions” are multiple choice questions that promote learning (rather than measurement). But it’s hard to know which ones work well, and which don’t. In partnership with an ed-tech startup (Eedi) we now have hundreds of millions of data points and are beginning to apply both statistics (Rasch analysis) and machine learning (variational autoencoders) to distilling insights from this data.

<hr/>

I fell in love with functional programming when I was 20, and I am still in love today. I helped design Haskell, and I spend a lot of time on GHC, Haskell's most widely-used compiler (including inside GitHub).



I got involved in education because I was dissatisfied with the computing education my children experienced. It has turned out to be much more rewarding than I expected, and I hope to share some of that excitement in my talk.

Simon Peyton-Jones

Principal Researcher

Computer Science Meets Education, at Scale

If you’ve ever wondered whether there is a better way to test software than by writing and maintaining thousands of test cases, then this talk is for you. I’ll explain “property-based testing”, which combines test case generation, test case simplification, and using properties to specify your code. I’ll show you what you need to think about to make it work in practice, and tell you tales of some of our more entertaining bugs.

<hr/>

I’ve loved functional programming all my career; I wrote “Why Functional Programming Matters”, a manifesto for the field, way back in the eighties. I urged lazy evaluation on John Backus (but failed to persuade him). I was a co-designer of Haskell. I moved to Sweden, home of the Lazy ML compiler, in the nineties, and at the turn of the millenium I co-invented property-based testing. I co-founded a startup in 2006 to bring PBT to industry, and I’ve learned to love (other peoples’) buggy software! My motto: “make hard things easy with functional programming”.

John Hughes

Experiences with QuickCheck: Testing the Hard Stuff


Online communities for programmers, like Stack Overflow and GitHub, have norms that are not obvious nor inclusive to the 50 million programmers visiting monthly. For example, many novices ask questions that go unanswered or downvoted for not conforming to unwritten community norms. In addition, the most popular online programming communities have reported having below 7% participation from marginalized developers such as women and non-binary people. But how do these norms and demographics shift for developers across the globe? Are their differences in which projects developers decide to contribute to? In this talk, I will 1) offer a perspective of understanding what can inhibit participation, 2) describe what mechanisms developers are using to alleviate their own frustrations with communities, and 3) outline what on-platform mechanisms can intervene to better support engagement based on rigorous research.

<hr/>

Dr. Denae Ford Robinson is a Senior Researcher in the SAINTes group at Microsoft Research where her research lies at the intersection of Human-Computer Interactions and Software Engineering. Her research identifies and dismantles cognitive and social barriers to software developer participation in online socio-technical ecosystems.

She obtained her Ph.D. in Computer Science and a graduate minor in Cognitive Science. She also holds an M.S. and B.S. in Computer Science from North Carolina State University. Throughout her Ph.D. academic career, she has interned and collaborated with research laboratories including MIT Lincoln Laboratory, Microsoft Research, Stack Overflow, and Facebook Research. Her dissertation research creates and applies a framework for dismantling barriers to participation titled “Identity-Based Signals and E-Mentorship to Support Engagement in Online Programming Communities”, which cites her impactful collaboration with Stack Overflow on the Mentorship Program.

Dr. Ford Robinson is also a recipient of the National GEM Consortium Fellowship, National Science Foundation Graduate Research Fellowship, and Microsoft Research Ph.D. Fellowship. She is also an Affiliate Assistant Professor at the University of Washington. More information about her research and publications can be found at https://denaeford.me.


CTO

Jason Warner

Speaker Series

Projects