Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it&#x27;s fantastic that it&#x27;s available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.

Ditto. So glad to see FoundationDB available again! Both the tech and the company have been missed. (Former FoundationDB Solutions Engineer :) )

Will said what I wanted to say, but: me too. I&#x27;m super happy about this and grateful to the team that made it happen!(I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it&#x27;s open source, I can rejoin the community!)

Your talk was one of the best talk i&#x27;ve seen , and i keep mentionning it to people whenever they ask me about distributed systems, database and testing.i&#x27;m incredibly impatient to have a look at what the community is going to build on top of that very promising technology.

The client is complex and needs very sophisticated testing, so there is only one implementation. All the language bindings use the C library.

Sorry if I missed it but can you or someone else please link where the client wire protocol is documented?

They are, slowly. Swift is open source, clang is open source. They are moving parts of the xcode IDE into open source, like with sourcekitd and now recently clangd.I don&#x27;t think they will ever move &#x27;secret sauce&#x27; into open source, but infrastructural things like DBs and dev tooling seems to be going in that direction.

~~How is it different from when Apple acquired the then-open-source FoundationDB (and shut down public access)? They could have just kept it open source back then.~~EDIT: My bad, looks like FoundationDB wasn&#x27;t fully open-source back then.

It is a genius move from Apple. I just wish they&#x27;d apply this logic to a lot of their other stuff.

I have nothing specific to add to this either other than I also spent years working there, and I&#x27;m unbelievably happy about this.

As an existence proof: before the acquisition we built an ANSI SQL database and a wire-compatible clone of the MongoDB API.I see no reason you wouldn&#x27;t be able to implement Datastore. In fact here&#x27;s a public source claiming that Firestore (which I believe is its successor) is implemented on top of Spanner: <a href="https:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2017&#x2F;10&#x2F;04&#x2F;google_backs_up_firebase_with_a_second_realtime_data_store&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2017&#x2F;10&#x2F;04&#x2F;google_backs_up_fir...</a>

Honest question, does MongoDB not work for this?

Hey there--I&#x27;m an engineer on the Cloud Datastore team. I&#x27;d love to know more about what your needs are if you&#x27;re willing to share.

I&#x27;m not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I&#x27;m trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.

A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos).In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.

I just watched the demo of 5 machines and 2 getting unplugged. The remaining 3 can form a quorum. What happens if it was 3 and 3? Do they both form quorums?

There were rumors iMessage moved there but ¯\_(ツ)_&#x2F;¯Would be great to see from Apple an engineering blog &quot;strengths and challenges at scale&quot; post now that it has been opened again.

&gt;It’s in widespread production use at Apple Anybody know if Apple migrated any projects from Cassandra to FoundationDB?Or was every project using FoundationDB a greenfield project?

Fantastic talk! Very engaging and insightful.

You don&#x27;t need to worry about shards &#x2F; partitions with FDB. Their transactions can involve any number of rows across any number of shards. It is by far the best foundation for your bespoke database.

It&#x27;s somewhat hard to answer your questions because the architecture (and hence, terminology) of FoundationDB is a little different than I think you are used to. But I will give it a shot.FoundationDB uses optimistic concurrency, so &quot;conflict ranges&quot; rather than &quot;locks&quot;. Each range is a (lexicographic) interval of one or more keys read or written by a transaction. The minimum granularity is a single key.FoundationDB doesn&#x27;t have a feature for indexing per se at all. Instead indexes are represented directly in the key&#x2F;value store and kept consistent with the data through transactions. The scalability of this approach is great, because index queries never have to be broadcast to all nodes, they just go to where the relevant part of the index is stored.FoundationDB delivers serializable isolation and external consistency for all transactions. There&#x27;s nothing particularly special about transactions that are &quot;cross-partition&quot;; because of our approach to indexing and data model design generally we expect the vast majority of transactions to be in that category. So rather than make single-partition transactions fast and everything else very slow, we focused on making the general case as performant as possible.Transaction coordination is pretty different in FoundationDB than in 2PC-based systems. The job of determining which conflict ranges intersect is done by a set of internal microservices called &quot;resolvers&quot;, which partition up the keyspace totally independently of the way it is partitioned for data storage.Please tell me if that leaves questions unresolved for you!

&gt; best known metadata storage engine for a distributed filesystem, hopsfsNot even close. I don&#x27;t even see anything I&#x27;d call a filesystem mentioned on your web page. I missed FAST this year, and apparently you had a paper about using Hops as a building block for a non-POSIX filesystem - i.e. not a filesystem in my and many others&#x27; opinion - but it&#x27;s not clear whether it has ever even been used in production anywhere let alone become &quot;best known&quot; in that or any other domain. I know you&#x27;re proud, perhaps you have reason to be, but please.

I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. 
When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have:
row-level locks
partition-pruned index scans
non-serialized cross-partition transactions (that is, a transaction coordinator per DB node)
distribution-aware transactions (hints on which TC to start a transaction on)The relative performance contribution of most of those features (cross-partition transactions not evaluated -it&#x27;s a must) can be seen in our paper at Usenix Fast:
<a href="https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;fast17&#x2F;fast17-niazi.pdf" rel="nofollow">https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;fast17&#x2F;fast17...</a>

That is so exciting! Can&#x27;t wait to poke around. I wonder if there&#x27;s anything in there I might recognize

&quot;we also simulate dumb sysadmins&quot;.
That was a really inspiring talk, thanks!

there were rumors that fdb had never been adopted at Apple because of some limitations and they still ran an in-house version of cassandra

Or perhaps it&#x27;s not so incredible?  Maybe it wasn&#x27;t such a huge hit for Apple and didn&#x27;t leave up to expectation so they figure they can give it away and earn some community goodwill.

It&#x27;s more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage.  FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.

How would you replace a Lucene&#x2F;Elasticsearch index with foundationDb?

They had a document layer that wasn&#x27;t released that was MongoDB compatible — so you can do this.

Yes you can. You need a tree index basically. Any kv store can serve as the backing data structure. I&#x27;ve been writing one for config file bidirectional transformation.

Would it be possible to build a tree DB on top of it like MonetDB&#x2F;Xquery? I always wondered why XML databases never took off, I&#x27;ve never seen anything else quit as powerful. Document databases if du jour seem comparatively lame.

If scale is a function of read&#x2F;writes, very large. In fact with relatively minimal (virtual) hardware it&#x27;s not insane to see a cluster doing around 1M writes&#x2F;second.

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16879392" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=16879392</a> massive

Apple also uses HBase for Siri I believe, what are some of the cluster sizes that FoundationDB scales to? Could it be used to replace HBase or Hadoop?I was in attendance at your talk, and thought it was one of the best at the conference. Apple I think broke some hearts completely going closed-source for a while, but glad to see them open sourcing a promising technology.

Where is the object-mapping-layer ? Inside the db in c++ or a service (different process etc) outside of it.

That&#x27;s impressive! Are you going to contribute your changes to foundationdb back?

I echo what wwilson has said. I work at Snowflake Computing, we&#x27;re a SQL analytics database in the cloud, and we have been using FoundationDB as our metadata store for over 4 years. It is a truly awesome product and has proven to be rock-solid over this time. It is a core piece in our architecture, and is heavily used by all our services. Some of the layers that wwilson is talking about, we&#x27;ve built them. metadata storage , object-mapping layer , lock manager , notifications system . In conjunction with these layers, FoundationDB has allowed us to build features that are unique to Snowflake. Check out our blog titled, &quot;How FoundationDB powers Snowflake Metadata forward&quot; [1]Kudos to the FoundationDB team and Apple for open sourcing this wonderful product. We&#x27;re cheering you all along! And we look forward to contributing to the open source and community.[1] <a href="https:&#x2F;&#x2F;www.snowflake.net&#x2F;how-foundationdb-powers-snowflake-metadata-forward&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.snowflake.net&#x2F;how-foundationdb-powers-snowflake-...</a>

Yes :) it&#x27;s the core of how Wavefront stores telemetry.

Not quite the same but TimescaleDB is Postgres-based and is showing a lot of promise for time-series data.

Know of any ideas around using this for long term Time Series data? Wonder if maybe something like OpenTSDB but with this as backend instead of hbase (which can be a sort of operational hell)

well FoundationDB for one doesn&#x27;t resort to clickbaity pseudo-benchmark marketing tactics<a href="https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;blog&#x2F;performance-part-two&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;blog&#x2F;performance-part-two&#x2F;</a>

How does this compare with CockroachDB? I&#x27;m planning to use CockroachDB for a project but would love to get an idea if I can get better results with FoundationDB.

&gt; Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale.<a href="https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;resiliency&#x2F;current&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;www.elastic.co&#x2F;guide&#x2F;en&#x2F;elasticsearch&#x2F;resiliency&#x2F;cur...</a><a href="https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;317-call-me-maybe-elasticsearch" rel="nofollow">https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;317-call-me-maybe-elasticsearch</a><a href="https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;323-call-me-maybe-elasticsearch-1-5-0" rel="nofollow">https:&#x2F;&#x2F;aphyr.com&#x2F;posts&#x2F;323-call-me-maybe-elasticsearch-1-5-...</a>The statement was blunt but the reputation is not exactly unearned. These problems become worse proportional to scale.

&gt; How about replacing your Lucene&#x2F;ElasticSearch index with something that actually scales and works?Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale.It&#x27;s definitely interesting but I&#x27;m cautious. The track record for FoundationDB and Apple has not been great here. IIRC they acquired the company and took the software offline leaving people in the rain?Could this be like it happened with Cassandra at Facebook where they dropped the code and then more or less stopped contributing?Also I haven&#x27;t seen many contributions from Apple to open-source projects like Hadoop etc. in the past few years. Looking for &quot;@apple.com&quot; email addresses in the mailing lists doesn&#x27;t yield a lot of results. I understand that this is a different team and that people might use different E-Mail addresses and so on.In general I&#x27;m also always cautious (but open-minded) when there&#x27;s lots of enthusiasm and there seems to be no downside. I&#x27;m sure FoundationDB has its dirty little secrets and it would be great to know what those are.

In reinforcement learning, the same approach (simulation) to addressing a data deficit is used, although the goal there is learning and not testing.When certain events happen rarely, you have fake them deterministically so that you have reproducible coverage.

&gt; In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4fFDFbi3toc" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4fFDFbi3toc</a>Unrelated to the original topic, but I had never come across that talk and it is great. I use the same basic approach to testing distributed systems (simulating all non-deterministic I&#x2F;O operations) and that talk is a very good introduction to the principle.

That presentation was really good! Well explained on the simulations. If one wanted to get into this exciting event and create something with FoundationDB but no database experience (I do know many programming languages) where would I start? If anyone could point me in the direction, I&#x27;d greatly appreciate it.

Long before we acquired Akiban, I prototyped a sql layer using (now defunct) sqlite4, which used a K&#x2F;V store abstraction as its storage layer. I would guess that a virtual table implementation would be similar: easy to get working, and it would work, but the performance is never going to be amazing.To get great performance for SQL on FoundationDB you really want an asynchronous execution engine that can take full advantage of the ability to hide latency by submitting multiple queries to the K&#x2F;V store in parallel. For example if you are doing a nested loop join of tables A and B you will be reading rows of B randomly based on the foreign keys in A, but you want to be requesting hundreds or thousands of them simultaneously, not one by one.Even our SQL Layer (derived from Akiban) only got this mostly right - its execution engine was not originally designed to be asynchronous and we modified it to do pipelining which can get a good amount of parallelism but still leaves something on the table especially in small fast queries.

How scalable and feasible to implement a SQL Layer on top of SQLite&#x27;s Virtual Table Mechanism (<a href="https:&#x2F;&#x2F;www.sqlite.org&#x2F;vtab.html" rel="nofollow">https:&#x2F;&#x2F;www.sqlite.org&#x2F;vtab.html</a>) which redirects the read&#x2F;write of the record data from&#x2F;to foundationdb?

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems.The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies.But the really interesting part is that it provides an extremely efficient and low-level interface for any other system that needs to scalably store consistent state. At FoundationDB (the company) our initial push was to use this to write multiple different database frontends with different data models and query languages (a SQL database, a document database, etc.) which all stored their data in the same underlying system. A customer could then pick whichever one they wanted, or even pick a bunch of them and only have to worry about operating one distributed stateful thing.But if anything, that’s too modest a vision! It’s trivial to implement the Zookeeper API on top of FoundationDB, so there’s another thing you don’t have to run. How about metadata storage for a distributed filesystem? Perfect use case. How about distributed task queues? Bring it on. How about replacing your Lucene&#x2F;ElasticSearch index with something that actually scales and works? Great idea!And this is why this move is actually genius for Apple too. There are a hundred such layers that could be written, SHOULD be written. But Apple is a focused company, and there’s no reason they should write them all themselves. Each one that the community produces, however, will help Apple to further leverage their investment in FoundationDB. It’s really smart.I could talk about this system for ages, and am happy to answer questions in this thread. But for now, HUGE congratulations to the FDB team at Apple and HUGE thanks to the executives and other stakeholders who made this happen.Now I’m going to go think about what layers I want to build…[1] Yes, yes, we ran Jepsen on it ourselves and found no problems. In fact, our everyday testing was way more brutal than Jepsen, I gave a talk about it here: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4fFDFbi3toc" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=4fFDFbi3toc</a>

Founder here. To the best of my knowledge, you are correct. ACID is getting thrown around a lot these days, and not being used in its strictest sense.

Google Spanner? Not OSS, but globally consistent, scalable, and extensively battle tested.

There is notThe speed and durability has gone unmatched. We can write millions of transactions per second w&#x2F; millions of reads w&#x2F;o issue. We have never lost data or found data to be inconsistent.

To the best of my knowledge (n.b.: I am not an expert, though I follow this field) there still aren&#x27;t any direct competitors to the breadth of what FoundationDB can do and do well.

cockroachdb and tidb&#x2F;tikv?
(= scalable key-value layer with transaction)

What competition do you feel FoundationDB has that has major traction and no lock-in?

If this was done years ago it&#x27;d be a big deal. Now I worry the competition will make this a non event.

Awesome - the code is up on github: <a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;</a>

My understanding of hbase (which could be wrong) is that writes of a particular key always go to the region master first, so if you read from the master you always get the latest value of a key. The tricky part is that when the region master goes offline another one needs to take its place, and you can get inconsistency or unavailability depending on how it is set up.I’d like to see a deep dive of how foundationdb handles this. It has to trade-off consistency for availability at some point and it would be nice to know exactly where.

An interesting fact is that Apple is probably one of the biggest users, if not the biggest user, of Cassandra out there.Can&#x27;t speak to HBase, but one thing Cassandra doesn&#x27;t guarantee is ACID - I&#x27;ve seen some data consistency issues that has arisen from Cassandra in our usage, although it hasn&#x27;t been a huge problem for us. That difference alone probably brings a lot of value to FoundationDB.

HBase and Cassandra both provide much weaker guarantees. At best they can support compare-and-set operations that are local to a single region&#x2F;&quot;row&quot;, whereas FoundationDB lets you do optimistic transactions (and consistent read snapshots) on the entire database. And at least in cassandra even getting that level of consistency comes at a big performance cost.The benefit of ACID transactions isn&#x27;t just safety at the application level, it&#x27;s the ability to compose abstractions and build complex data models on top of simple ones. For example, indexes in FoundationDB are typically more scalable than in the majority of other systems, where index queries often have to be broadcast to all the systems in a cluster. Yet FoundationDB doesn&#x27;t even have indexes as a feature - higher layers build and maintain index invariants using transactions.FoundationDB is also just really reliable and fault tolerant. Its testing story is drastically better than what the teams building these other products are doing.

How does this stack up against HBase and Cassandra, which seem to have gotten traction already in the same areas that FoundationDB seems best suited for?

A lot of people apparently hated that I attacked this person, and downvoted me; but what the person who posted this comment was doing is intentionally throwing shade at and casting doubt on a project by saying &quot;I think that the claims on this webpage don&#x27;t make any sense as this should be impossible&quot;. It comes off as &quot;oh come on, this doesn&#x27;t even sound plausible; I call bullshit&quot;.This then causes people who aren&#x27;t versed with the product or the technology to decrease their perception of the product, and puts the team behind it in a position of having to not just come to its defense but to do so quickly due to the perception concerns.Meanwhile, if they just do a basic search for &quot;MVCC serializable&quot; they would find that they were wrong; which means that it took more time to leave this insulting comment than it would have taken them to learn how this can work.As a community, we really really really really need to beat down on casual cynics like this, who like to lazily &quot;call bullshit&quot; or play the &quot;citation needed&quot; card as a way to undermine the credibilty of other peoples&#x27; products. We live in a future where the answers to these kinds of doubts are a moment away: this particular form of debate tactic needs to die.

I mean, PostgreSQL is MVCC and it clearly supports serializable as an isolation level... have you tried using Google?<a href="https:&#x2F;&#x2F;wiki.postgresql.org&#x2F;wiki&#x2F;Serializable" rel="nofollow">https:&#x2F;&#x2F;wiki.postgresql.org&#x2F;wiki&#x2F;Serializable</a>

Thanks, that explanation is really helpful!

Great write-up.Is this similar to how Software Transactional Memory (STM) is implemented? It sounds very very similar indeed.

Thanks. Can you elaborate on how 6 is actually accomplished? Various earlier comments have hinted that the transactional authority (conflict checking) can actually scale &#x27;horizontally&#x27; beyond the check-throughput that can be archived by a single node. Is that the case? and whats the magic sauce for doing that for multi-object transactions? :)

This is a good explanation of how it happens on a single node. What do you do when the transaction is distributed? How do you achieve consensus? Is there a write up on it anywhere?

Does the implementation handle the case that you want to do a write that is conditioned on a prior read finding no corresponding record(s)?

Any chance you’d be able to find that video, it sounds extremely interesting.

I&#x27;ll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency.A FDB transaction roughly works like this, from the client&#x27;s perspective:1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction2. Do reads from a consistent MVCC snapshot at that read version. No matter what other activity is happening you see an unchanging snapshot of the database. Keep track of what (ranges of) data you have read3. Keep track of the writes you would like to do locally.4. If you read something that you have written in the same transaction, use the write to satisfy the read, providing the illusion of ordering within the transaction5. When and if you decide to commit the transaction, send the read version, a list of ranges read and writes that you would like to do to the distributed database.6. The distributed database assigns a write version to the transaction and determines if, between the read and write versions, any other transaction wrote anything that this transaction read. If so there is a conflict and this transaction is aborted (the writes are simply not performed). If not then all the writes happen atomically.7. When the transaction is sufficiently durable the database tells the client and the client can consider the transaction committed (from an external consistency standpoint)The implementations of 1 and 6 are not trivial, of course :-)So a sufficiently &quot;slow client&quot; doing a read write transaction in a database with lots of contention might wind up retrying its own transaction indefinitely, but it can&#x27;t stop other readers or writers from making progress.It&#x27;s still the case that if you want great performance overall you want to minimize conflicts between transactions!

&gt; I didn&#x27;t think an MVCC implementation could provide that level of isolationMVCC needs a bit of additional logic ontop to be serializable - &quot;Serializable snapshot isolation&quot; is a good 
keyword to search for - But it&#x27;s definitely possible.<a href="https:&#x2F;&#x2F;courses.cs.washington.edu&#x2F;courses&#x2F;cse444&#x2F;08au&#x2F;544M&#x2F;READING-LIST&#x2F;fekete-sigmod2008.pdf" rel="nofollow">https:&#x2F;&#x2F;courses.cs.washington.edu&#x2F;courses&#x2F;cse444&#x2F;08au&#x2F;544M&#x2F;R...</a><a href="https:&#x2F;&#x2F;drkp.net&#x2F;papers&#x2F;ssi-vldb12.pdf" rel="nofollow">https:&#x2F;&#x2F;drkp.net&#x2F;papers&#x2F;ssi-vldb12.pdf</a><a href="https:&#x2F;&#x2F;wiki.postgresql.org&#x2F;wiki&#x2F;Serializable" rel="nofollow">https:&#x2F;&#x2F;wiki.postgresql.org&#x2F;wiki&#x2F;Serializable</a>Edit: Formatting

Apache Trafodion does as well:<a href="http:&#x2F;&#x2F;trafodion.apache.org" rel="nofollow">http:&#x2F;&#x2F;trafodion.apache.org</a>

I hadn&#x27;t heard of FoundationDB before, so I did some digging into the features: <a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;features.html" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;features.html</a> . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won&#x27;t slow down operations, and that it allows true interactive queries. I didn&#x27;t think an MVCC implementation could provide that level of isolation, and I&#x27;m not even sure how you provide that level of isolation and those other guarantees with any implementation, am I misunderstanding something?

Swift, too, was published on GitHub including its whole version control history, dating back to 2010 :)

It&#x27;s under Apache 2.0 for those curious: <a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;LICENSE" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;LICENSE</a>. Also, side note: it looks like this was a private GitHub repository for at least a couple months, since they have pull requests going back for at least that long. I find this interesting, since Apple normally &quot;cleans up&quot; history before open sourcing.

This is amazing! Is there a roadmap of what is being worked on or is this mainly a dump of their existing codebase?From looking at the commit history it seems like this is pretty actively developed.

iCloud is an umbrella term for several different systems.

I had always assumed they acquired the team more than the ip. I&#x27;m not sure this confirms that or not. I&#x27;d be curious what the answer to this question is as well, but wouldn&#x27;t be surprised if the answer was &quot;they don&#x27;t&quot;

CloudKit is using Cassandra&#x2F;Solr:- <a href="http:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol11&#x2F;p540-shraer.pdf" rel="nofollow">http:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol11&#x2F;p540-shraer.pdf</a>

Maybe CloudKit? I was looking at the documentation for it recently and it&#x27;s a key-value store.

I&#x27;m confused. How they could use FoundationDB when Apple aquired it a long time ago? Did apple sell the software to other companies?

Wavefront&#x27;s co-founder just tweeted this: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;panghy&#x2F;status&#x2F;987022825457266689" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;panghy&#x2F;status&#x2F;987022825457266689</a>Finally it&#x27;s out! @WavefrontHQ managaes petabyte scale clusters with #foundationdb today!

Do we know where&#x2F;how Apple uses FoundationDB in production ?

this is amazing and this is huge for us - FoundationDB is one of the fastest most scalable distributed KV stores.

Caché is more oriented towards vertical scaling, big monolithic servers, where FoundationDB is more towards horizontal scaling.Also, the obvious difference is that Caché is closed source and prohibitively expensive.FiS is only FOSS on selected platforms (Linux x86, OpenVMS Alpha), and proprietary on all other platforms.

Yeah, when I read about that I thought it sounded neat—make sure the index updates with the data. On the other hand, I think of CAP theorem as an iron triangle. If you are gaining consistency, what’s the trade off?

It seems that MUMPS doesn&#x27;t have serializable isolation.

Oh, Cache...Worked a job once where that was the underlying data store.I was only allowed to touch the SQL interface to it, which was....weird.The SQL dialect was ancient, felt like something from about 1990 (and this was in.... 2012 or so, so not THAT long ago).Query performance seemed invariant. A simple select * from foo where id=X and a monster programatically generated join across 15 tables would both take about 1.5 seconds to return results.

&quot;because it is an ordered key-value store, FoundationDB can use range reads to efficiently scan large swaths of data&quot;<a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;features.html" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;features.html</a>I wonder how it compares to MUMPS databases like Intersystems Cache and FIS GtM?

The storage engine has always been SQLite for the SSD engine.

I don&#x27;t think that is correct. My understanding from using fdb for years in production was that sqlite was the storage engine.

Truthfully at Wavefront we&#x27;ve taken the json status directly into telegraf. Plus a bunch of python tooling to massage additional telemetry on a clusters health (coordinator reachability for example).Plus even more tooling (mostly Ansible) for managing large fleets.

The storage engine is and always was a fairly heavily modified asynchronous version of sqlite&#x27;s btree. It&#x27;s been extremely reliable, which was always our top priority, and the performance isn&#x27;t bad. But honestly when there was a problem with it our development velocity improving it wasn&#x27;t great.It&#x27;s super easily pluggable[1], so now that it is open source people can experiment with other engines. I think there is a lot of room for improvement. Also architecturally it&#x27;s designed in anticipation of being able to run different storage engines for different key ranges and for different replicas. For example, you might keep one replica in a btree on SSD (for random reads) and two on spinning disks in a log structured engine.[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;fdbserver&#x2F;IKeyValueStore.h" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;fdbserver&#x2F;...</a>It looks to me like Apple has made a pretty complete release of the key&#x2F;value store. What&#x27;s missing is(1) Layers! Everything from relational databases to full text search engines to message queues(2) Monitoring stuff. Unsurprisingly it doesn&#x27;t look like we have the tools for monitoring log files, etc. Wavefront (also a major user!) is a great commercial solution, but there should be something OSS

This is great news!Unfortunately it looks like they striped out some important things, notably the storage engine (there&#x27;s now a sqlite fallback).Edit: Apparently it was always sqlite as per replies bellow.

Fair enough, I stand corrected.  Superficially it seems like a more pleasant experience than dealing with gn&#x2F;ninja as an end user.

Ah, GNU make second expansion... glad to see someone else appreciates it ;-).

This kind of deep-inside-baseball from-the-horses-mouth interaction is what&#x27;s so awesome about HN!

Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build.Although it starts by running `make`, it&#x27;s about as in-house as a thing can be.

Google please take a few notes here:1. It’s in its own repo2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links.3. They use a common build system and not an in-house thing.

<a href="https:&#x2F;&#x2F;www.foundationdb.org&#x2F;download&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.foundationdb.org&#x2F;download&#x2F;</a>

I’m sure one will be shortly forthcoming. They did a good job up actually putting up the code quickly following (in parallel) with the announcement.

Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don&#x27;t provide a distribution that will work on it.  I know the license says Apple doesn&#x27;t have to do anything, but it just seems wrong that they didn&#x27;t provide a download.

Ahah! That thread&#x27;s so sad vs. this thread&#x27;s so exciting :)

Original thread for when Apple acquired FoundationDB for those curious: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9259986" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9259986</a>

Well for one, Spanner is not available except as a service.

Wait how can it be ACID if it can tolerate N failures from N+1 nodes? Doesn&#x27;t that kill consistency almost by definition? What level of isolation does it support? Snapshot? Serializable?

The datacenter-aware mode documentation [0] says “Although data will always be triple replicated in this mode, it may not be replicated across all datacenters.”Why is that?[0] <a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;configuration.html?datacenter-aware-mode" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;configuration.html?data...</a>

&quot;or example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance&quot;Wait, don&#x27;t you need 3N+1 to tolerate N number of failures for it to be Byzantine fault tolerant? Is that not a goal of FoundationDB?

I think Citus is not really ACID.Spanner (and to an extent its less mature OSS descendants Cockroach and TiKV) has more comparable goals, but is fairly different architecturally. For example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance (even lots of databases with much weaker guarantees are in the latter category!), doesn&#x27;t trust clocks at all, doesn&#x27;t lose performance when transactions cross replica sets, and uses optimistic instead of pessimistic concurrency.Also FoundationDB (and TiKV) make a distributed, transactional key&#x2F;value store available as an API, while Spanner and Cockroach expose only a relational database layer. FoundationDB is designed philosophically with the idea that you want to have a single storage layer to manage operationally but should be able to mix and match data models and query engines above that layer.On the other hand, FoundationDB doesn&#x27;t currently have any full fledged high level database layer available. Someone will probably dig up our SQL layer (which was AGPL, I think) but I wouldn&#x27;t really recommend using it in production because there is no active development team. Someone will probably try porting the SQL layers from TiDB and Cockroach.Maybe Apple will open source more stuff in the future, but let&#x27;s not get too greedy!

CockroachDB (and I assume TiDB as well) can have the same properties and same awesome latencies as Spanner, if you have the same hardware.So if you actually put down your own fiber, and install GPS clocks in each rack, you’ll be able to enjoy the same results.

It&#x27;s more like what CockroachDB or TiDB use underneath. They all are suited for local clusters, but cannot perform well enough replicating across multiple far away datacenters over public internet, latency trade offs would be unbearable. Spanner is a bit different, with Google&#x27;s fancy clocks and fancy networks, it can get better latency trade offs that might satisfy more applications.Citus I can&#x27;t remember, but if Citus does it properly wrt consistency, it should be in the same category as CockroachDB and TiDB, there is no magic.

Spanner is geographically distributed and synchronizes over Google’s fiber. I’m not sure where FoundationDB falls on the spectrum, but there is “distributed” meaning runs on multiple machines, vs multiple availability zones (which could be in the same rough geographic area to minimize latency), vs multiple geographic regions, which is one of Google’s design goals for its distributed infrastructure nowadays: to be able to withstand a natural disaster or other disruption that takes a whole region offline, without missing a beat.There’s still a lot to be said for the property, “scales horizontally to N machines,” of course, even if those machines have to be in the same data center.

I hate to be that person, but when I hear &quot;ACID transactions in a distributed database&quot;, I hear Citus&#x2F;Spanner&#x2F;CockroachDB.I&#x27;m positive that Citus &amp; Spanner are quite different from FoundationDB, but I have no idea how. Googling didn&#x27;t help much.Can someone provide an overview of the differences?

Looks like Safari will happily load a pdf in an img tag, but Chrome won&#x27;t

It&#x27;s a pdf: <a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;_images&#x2F;Architecture.pdf" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;_images&#x2F;Architecture.pd...</a>

Broken documentation: Architecture image is missing. <a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;architecture.html" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;architecture.html</a>Hope someone from the team reads this.

That&#x27;s a great school.  I hired some students there when I ran a tech camp for middle-schoolers and they were...beyond.

I did chuckle at the part above about the &quot;best software engineers in high school,&quot; but can&#x27;t argue with the results.

Just checked the rankings. My first high school made #172 on the list (&quot;International School of the Americas&quot;).My second high school is unranked (&quot;Texas Academy of Math and Science&quot;). I don&#x27;t think it qualifies as a high school. Seems we haven&#x27;t done a great job of identifying the accomplishments of our alumni, based on the Wikipedia page. No doubt it would rank near the top though. My year alone Caltech accepted about 30 of us, more than any other high school in the country. Makes me wonder what my peers have been up to.Anyway, I&#x27;d agree that these tech high schools have some amazingly smart people attending them.2: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Texas_Academy_of_Mathematics_and_Science#Notable_alumni" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Texas_Academy_of_Mathematics_a...</a>

The top high schools in the Bay Area often give it a run for its money…

I went to UVA and I think about half of the engineering school was from TJ (including 3 of my roommates). :)  I can&#x27;t think of any superior public high school (and I went to a different Governor&#x27;s School in Virginia myself) anywhere, due to the amazingly large and deep talent pool TJ pulls from.  Nothing like it exists in the Bay Area, that&#x27;s for sure!

Forgot to mention that those four engineers were same school AND same grade as me.  So 4 out of 400 students were the tech guys behind Yext and FoundationDB.

Did you bow down at your laptop?Is it ok to admire great engineers on HN?Is this comment relevant to anything but your own inferiority complex?Can we get a decent definition of hubris in here?

Maybe it&#x27;s just me, but the hubris in this comment seems a little excessive.They aforementioned SWEs make themselves multimillionaires &#x2F;and&#x2F; have great jobs &#x2F;and&#x2F; get praised by their peers, yet now everyone else has to bow down to them... over claims they were great programmers in high school? Is an appeal to your own experience the best way to make yourself seem relevant here?It&#x27;s an incredible DB system, but it ain&#x27;t the Second Coming. Calm down.

I went to the same high school as the founders[1]. They were about the 2 best software engineers in a school with a LOT of very smart software engineers. Another pair founded Yext, which went public last year. I still consider that school the group with the highest concentration of raw brain power I&#x27;ve ever been a part of.I&#x27;m probably a 1% engineer, been hired by M$, FB, and Google. These guys were light years ahead of me. I&#x27;m not sure I&#x27;m as good now as they were at like 17 years old. In fact I&#x27;m probably only a decent engineer from having observed the stuff they were doing back then and finding inspiration.1: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Thomas_Jefferson_High_School_for_Science_and_Technology" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Thomas_Jefferson_High_School_f...</a>

Agreed, technology-wise FDB and TIDB look almost the same.

Well on first guess, I would assume that FoundationDB is more mature than tikv, although I have never used either but tikv looks cool

Here&#x27;s a brief comparison by a tikv developer: <a href="https:&#x2F;&#x2F;github.com&#x2F;pingcap&#x2F;tikv&#x2F;issues&#x2F;2986#issuecomment-383355986" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pingcap&#x2F;tikv&#x2F;issues&#x2F;2986#issuecomment-383...</a>

How&#x27;s this compare to <a href="https:&#x2F;&#x2F;github.com&#x2F;pingcap&#x2F;tikv" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pingcap&#x2F;tikv</a>? It&#x27;s a relatively new distributed KV store written in Rust that also is transactional and backs the new TiDB database.

I don&#x27;t recall them ever having separate processes by default. You can set that up though by limiting roles.

Wait CockroachDB is monolithic? I though it was distributed.

&gt; monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibilityThe CockroachDB documentation says their SQL implementation is layered on top of their K&#x2F;V interface: <a href="https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;docs&#x2F;stable&#x2F;architecture&#x2F;overview.html#layers" rel="nofollow">https:&#x2F;&#x2F;www.cockroachlabs.com&#x2F;docs&#x2F;stable&#x2F;architecture&#x2F;overv...</a>This would make it similar to TiDB&#x2F;TiKV and FoundationDB.Here&#x27;s and old article from VoltDB insisting that layering SQL on pure K&#x2F;V forfeits too much performance: <a href="https:&#x2F;&#x2F;www.voltdb.com&#x2F;blog&#x2F;2015&#x2F;04&#x2F;01&#x2F;foundationdbs-lesson-fast-key-value-store-not-enough&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.voltdb.com&#x2F;blog&#x2F;2015&#x2F;04&#x2F;01&#x2F;foundationdbs-lesson-...</a>Edit: And a response: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15505194" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15505194</a>

It&#x27;s closest to TiDB&#x27;s key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling.In my understanding, FoundationDB&#x27;s transaction management is closest to FaunaDB&#x27;s; read&#x2F;write sets are linearized in memory in preprocessing nodes and distributed asynchronously to the replicas rather than locked on the replica leaders like Spanner or CockroachDB. This is why FoundationDB doesn&#x27;t support long-lived transactions.It&#x27;s interesting that the FoundationDB team chose to unwind their service architecture (there used to be separate transaction manager and replica processes), I assume in the interests of ease of operations.It is not clear to me how leader election and failover works for the transaction management role. Maybe somebody from the team can clarify.

How does it compare to CockroachDB or TiDB?

Anyone wants to give a summary of what primitives&#x2F;structures&#x2F;algorithms FoundationDB uses?

Does anyone know if the SQL layer was open sourced, or whether there are any plans to open source it?

Wow this is super exciting. I had a half dozen things that I though FDB would be good for and then poof! it got sucked up into the Apple spaceship. Now to have it emerge unscathed is pretty awesome.Another step closer to a library appliance.

How is FoundationDB&#x27;s graph performance&#x2F;feature set compared to those of other graph and multi-model databases?

The core product is a distributed, highly fault tolerant ordered Key Value store with true serializable ACID transactions. All of the layers (including document, graph) sit on top of that and inherit its ACID properties, scalability, fault tolerance, etc. It doesn&#x27;t appear to me that they released any of the top level layers, but those are MUCH simpler to build, and that&#x27;s where the OS community can step in.

Originally, FDB was a DB supporting 3 models:
- KV
- Document
- GraphIt seems the announcement concerns only the KV one. Someone has information for the 2 other ones?Thank you.

You should have seen their trade show demos, where they&#x27;d demonstrate fault tolerance by literally powering down random computers and unplugging network cables, while visualizing how the system managed fault tolerance.ah found it: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=oLGYMdo2q2g" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=oLGYMdo2q2g</a>

I can see that. There are various vectors to performance. I hear you saying transactions per second would be unimpressive.A couple of less time sensitive applications are: 1. distrubuting information the entire fleet should eventually know, 2. event log aggregation with fine-grained time alignment among nodes.Both are probably silly problems to solve with a database, killing houseflys with sledghammers and all that, but it never hurts to explore creative tool misuse :)

The fault tolerance is pretty much flawless. You won&#x27;t be able to get the database &quot;stuck&quot; or see anomalies.But performance is going to suck if you run server nodes over unreliable connections. I have trouble seeing a FoundationDB cluster running on mobile robots as more than a trade show gimmick. Albeit an awesome gimmick. So in summary you should totally do that.

Question for the FoundationDB gurus that are hanging out on this thread: How well does it deal with spotty connectivity? I&#x27;m asking because I work on mobile robots, and WiFi and&#x2F;or LTE connections are always coming and going in unpredictable ways as the vehicle moves about it&#x27;s environment.  Reconnecting every few minutes is normal.

I found out foundationDB vienna VA office is closed.
Did apple move all people to Cupertino?

also super interested in CockroachDB, but I just can&#x27;t find enough war stories, or stories of people using it in production...

Firstly: Wow! this is amazing news!!!I&#x27;m also kind of confused.. is the single repo complete?What about the SQL Layer [0]? Where is all this stuff in the new GH repo?Or is only the KV part be open-source?Looking forward to some CockroachDB vs. FDB benchmark showdowns :)[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;jaytaylor&#x2F;sql-layer" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jaytaylor&#x2F;sql-layer</a>

Does it mean that client is unusable on Windows?

Nope, the actor compiler is down with mono and your need windows to compile the windows client. When we were independently working on it, we just ditched the windows code completely (we are a mac&#x2F;linux shop)

Before FDB was acquired by Apple, a lot of the engineers used Visual Studio on Windows. It&#x27;s a good IDE for C++. You certainly don&#x27;t need it to develop though!Windows was never the most important deployment platform for the product. I&#x27;m not sure how performant the server is. But it should work, and I would think the client is fine.

Noticed the Visual Studio project files. Was Windows a primary development platform for foundationdb?

Not that this really has anything to do with FoundationDB, but why do you say that object storage is a poor substrate for file and block abstractions? There are many high-performance block and file systems built on object storage.

Absolutely right - I wrote an object storage backed filesystem for Hadoop and synchronizing was a complete nightmare.It certainly worked well within our required architecture - but as a general purpose system it would have several issues over typical network topologies.As active volume size increases and&#x2F;or changes, the object storage layer latency can bump up to minutes, or even hours. We stopped tested at 100TB volumes, as the object storage layer was backed up to hours.Object Storage is very convenient, but the lack of good metadata and latencies involved basically means that it&#x27;s only good as an archival backing store of active data. If your active datasource goes out - you could lose hours of data unless you have local copies.

There are several distributed storage systems like Ceph and they all have problems. Ceph is not good because it&#x27;s an object storage system trying to provide block storage and a filesystem on top, which will never work well.

These days most people know Hadoop for a distributed storage. In my opinion, though CEPH [1] has the bigger potential.[1] <a href="https:&#x2F;&#x2F;ceph.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ceph.com&#x2F;</a>

I also wonder the performance when concurrency writes to a few   keys.

is Transaction authority in-memory or on disk?
this architecture seems kinda clunky. Wondering what&#x27;s the performance and is this a good fit for quick k-v store use cases?

I remember reading about Apple purchasing FoundationDB and everyone being disappointed and hoping they&#x27;d open source it. And now they have!

We run both, memory engine hasn&#x27;t seen too many updates in the last 3 years though. For SSDs, we use i3 as well as EBS-backed volumes.

Found this section on performance very interesting.<a href="https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;performance.html#throughput-per-core" rel="nofollow">https:&#x2F;&#x2F;apple.github.io&#x2F;foundationdb&#x2F;performance.html#throug...</a>I wonder what the SSD engine performance would look with NVMe standard NAND or an Optane SSD instead of SATA. Any FoundationDB guys&#x2F;gals on this thread able to comment?Another Q: what&#x27;s more commonly used in current FoundationDB deployments: memory engine or storage engine?

&gt; I remember all the people who bashed Apple when they acquired FoundationDB.I&#x27;m not ashamed about deriding apple for Apple taking a really, really great product and hiding it from the world for years to come.This is definitely some atonement, but does not totally absolve Apple from the many times they&#x27;ve taken tech private.

This sounds like a business opportunity to me rather than a cautionary note.I remember all the people who bashed Apple when they acquired FoundationDB. I hope they are appropriately ashamed now.

&gt; IMO In the stateful world, operating&#x2F;running the damn thing is 85% of the work, 15% being active dev. (Stateless world is a little better, but still painful).The number of newbie engineers who see docker&#x2F;kubernetes, and think &quot;let me docker run or helm install&quot; a stateful service in a couple of minutes - is mind boggling.I&#x27;ll remember this quote when trying to talk sense to them.

This is great news, when I was with dynamo, FoundationDB was the other green shore for me :). They did so many things so well.A tiny bit of caution for folks trying to run systems like this though: It is frigging hard at any reasonable scale. The whole thing might be documented &#x2F; OSS and what not, but very soon you are going to run into deep enough problems that&#x27;s going to require very core knowledge to debug, energy to deep dive. Both of which you probably don&#x27;t want to invest your time into. Do evaluate the cloud offerings &#x2F; supported offerings before spinning these up. Else ensure you have hired experts who can keep this going. They are great as a learning tool, pretty hard as an enterprise solution. I have seen the same issue a ton of times with a bunch of software (redis&#x2F;kafka&#x2F;cassandra&#x2F;mongo...) by now. IMO In the stateful world, operating&#x2F;running the damn thing is 85% of the work, 15% being active dev. (Stateless world is a little better, but still painful).

This is such a great news. I had seen, FoundationDB guys present in new england database summit couple of years ago.  Their demo reminded me of sun microsystems famous demo of screwing their hard disk.

Is it known whether Apple runs this code in production?

Cassandra allows you to configure a &quot;partitioner&quot; that determines which nodes a primary key belongs to. There is a ByteOrderPartitioner that stores partitions in order lexicographically by primary key. There is also a Murmor3 hash based partitioner (which is the recommended default).Cassandra allows you to store multiple records in sorted order within a partition. The normal recommended way to get data locality is to store records that are frequently accessed together in the same partition.

Does Cassandra hash the primary key to get a more even distribution?If you have a sorted data store, you can get the same distribution by keying off a hash of the &quot;real&quot; primary key, right?

I&#x27;m very interested in hearing more about what running FoundationDB in production is like.I believe that FoundationDB stores rows in lexicographical order by key. Other databases like Cassandra strongly push you toward not storing data this way as it can easily lead to hotspots in the cluster. How do you deploy a FoundationDB cluster without leading to hotspots, or perhaps what operational actions are available to rebalance data?

Also, Windows Defender SmartScreen complains about unknown publisher when trying to execute the installer.

The Windows installer has Lorem Ipsum instead of EULA text... <a href="https:&#x2F;&#x2F;twitter.com&#x2F;stijnsanders&#x2F;status&#x2F;987042633691394050" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;stijnsanders&#x2F;status&#x2F;987042633691394050</a>

Is there a canonical php client for this?

comdb2 is not a big data db per say. it compares more to mysql than foundationdb 
<a href="https:&#x2F;&#x2F;blog.dripstat.com&#x2F;first-look-at-bloombergs-amazing-comdb2&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blog.dripstat.com&#x2F;first-look-at-bloombergs-amazing-c...</a>

It’s neat although there is no sql front end.Bloomberg’s comdb2 was open sourced recently <a href="https:&#x2F;&#x2F;github.com&#x2F;bloomberg&#x2F;comdb2" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;bloomberg&#x2F;comdb2</a> - it seems similar, but would be interesting to see comparison.

Well, it&#x27;s infrastucture. Moreover, it&#x27;s infrastructure for infrastructure! So if that sounds super boring you don&#x27;t have to be excited :-)But I would tell the story something like this: state storage is the root of (almost) all operational evil. It&#x27;s very easy to make a system reliable if it&#x27;s totally stateless. Even most bugs can be lived with if the worst you have to do is restart a service and carry on! But to do anything interesting you have to store state somewhere, and you have to modify that state concurrently without screwing it up.And the many challenges of operating stateful systems are greatly multiplied if you have a lot of different ones. For example, if you have a datacenter outage and some but not all of your stateful systems deal with it correctly, probably your application as a whole is still down.So as one more stateful system, does FoundationDB just make that worse? Well, FoundationDB is designed specifically to be a foundation for many very different stateful systems - not just different kinds of databases but things like search engines or message queues that you normally don&#x27;t think of in the same category. So that almost any system can map to it efficiently, it has a lowest common denominator data model (key&#x2F;value) and the highest possible guarantees in terms of concurrency control. And you can run diverse systems supporting an application on the same FoundationDB cluster, or on different clusters with the same exact operational requirements.Some few users of FoundationDB have been able to get the benefits of this vision, consolidating lots of different stuff into a single, operationally desirable system. But for more people to be able to, not just does the key&#x2F;value store have to be available to them, but also lots of stuff has to be built on top of it. By releasing FoundationDB under a very liberal open source license, Apple has hopefully made that possible. In the long run, hopefully it will make all server-side computing more reliable.Also, it&#x27;s a really good key&#x2F;value store, if you happen to need one of those!

Can someone explain to business &#x2F; product guy why this is so exciting? (honestly keen to understand)

For those asking about use in production, Wavefront just posted this:&quot;Wavefront by VMware’s Ongoing Commitment to the FoundationDB Open Source Project&quot;<a href="https:&#x2F;&#x2F;www.wavefront.com&#x2F;wavefront-foundationdb-open-source-project&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.wavefront.com&#x2F;wavefront-foundationdb-open-source...</a>

&quot;but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS &quot; &lt;=== This I wish more people realized that is a day to day reality if you are in AWS at scale.

We basically replaced mySQL, Zookeeper and HBase with a single KV store that supports transactions, watches, and scales. It&#x27;s not a trivial point that you can just develop code against a single API (finally Java 8 CompletableFutures) and not have to set up a ton of dependencies when you are building on top of FDB. We are (obviously) experts at monitoring FoundationDB with Wavefront and we hope to release the metric harvesting libraries and template dashboards that we use to do so.Almost 5 years in and we have not lost any data (but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS =p).

We (Wavefront) has been operating petabyte scale clusters for the last 5 years with FoundationDB (we got the source code via escrow) and we are super excited to be involved in the opensourcing of FDB. We have operated over 50 clusters on all kinds of aws instances and I can talk about all the amazing things we have done with it.<a href="https:&#x2F;&#x2F;www.wavefront.com&#x2F;wavefront-foundationdb-open-source-project&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.wavefront.com&#x2F;wavefront-foundationdb-open-source...</a>

Wow this is some really exciting news! I think it would be amazing to create a GraphQL API for FoundationDB. Therefore i have created a feature request for this in the Prisma repo. For those who don&#x27;t know Prisma is a GraphQL database mapper for various databases.<a href="https:&#x2F;&#x2F;github.com&#x2F;graphcool&#x2F;prisma&#x2F;issues&#x2F;2240" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;graphcool&#x2F;prisma&#x2F;issues&#x2F;2240</a>

Answered my own question (seems like the answer is yes):<a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;bindings&#x2F;python&#x2F;fdb&#x2F;impl.py#L1584" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;master&#x2F;bindings&#x2F;p...</a>

Does anyone know if the python API for fdb works with asyncio?

Does anyone know what services at Apple are built on top of FoundationDB?

More info on this in this issue: <a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;issues&#x2F;129" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;issues&#x2F;129</a>

Found the NodeJS bindings in the 5.0 branch: <a href="https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;release-5.0&#x2F;bindings&#x2F;nodejs&#x2F;package.json.in" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;apple&#x2F;foundationdb&#x2F;blob&#x2F;release-5.0&#x2F;bindi...</a>Seems dated as it requires NodeJS either 0.8 or 0.10.

Why did it take three years? Apple killed FoundationDB as a product in March 2015.

Seconding this. Instead of hearing from bloggers or contributors, what is using this DB like in the trenches? What simple use case would fit this DB perfectly? Is there a lot of setup on second&#x2F;third deploy? Easy to maintain, or requires a lot of tweaking&#x2F;tuning? How is memory usage with 10M records? 100M?

It probably is better, because Apple switched from Cassandra to FoundationDB, according to all the rumors <a href="http:&#x2F;&#x2F;www.businessinsider.com&#x2F;why-apple-bought-foundationdb-2015-3?IR=T" rel="nofollow">http:&#x2F;&#x2F;www.businessinsider.com&#x2F;why-apple-bought-foundationdb...</a> . But as we know Apple they probably won&#x27;t tell us.

Personally I&#x27;d be more interested in hearing how this compares to other distributed noSQL implementations like Cassandra.

I can see everyone&#x27;s extremely happy about this, which is great. As someone who&#x27;s never used it, I&#x27;d like to know more about FoundationDB and how it compares to other offerings such as MySQL or Postgres, and which use cases is it most suited to. I would especially love to hear the thoughts of those with direct experience of using Foundation DB. Thanks!

That&#x27;s from sqlite. (Awesome tech, awesome license).Related snippet from the &quot;Distinctive Features Of SQLite&quot; page[1] from the sqlite project:The source code files for other SQL database engines typically begin with a comment describing your legal rights to view and copy that file. The SQLite source code contains no license since it is not governed by copyright. Instead of a license, the SQLite source code offers a blessing:May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.[1] <a href="https:&#x2F;&#x2F;sqlite.org&#x2F;different.html" rel="nofollow">https:&#x2F;&#x2F;sqlite.org&#x2F;different.html</a>

May you do good and not evil.
 May you find forgiveness for yourself and forgive others.
 May you share freely, never taking more than you give.from their source code blessing. notes.

In addition to single-key asynchronous watches, there are also versionstamped ops (for maintaining your own, sophisticated log in a layer) and configurable key range transaction logging (but see the caveats in my other post on the topic).I&#x27;m not sure it has every feature it will ever need in this area, but it&#x27;s a pretty good starting point for building &quot;reactive&quot; stuff.

It has the ability to watch keys so building a notification system on top of it is pretty easy. Really only limited by your imagination.

Looks promising, but does anyone know if FoundationDB has external events or triggers, similar to Firebase or RethinkDB? I can&#x27;t seem to find much on it.If not, then a lot of potential is being left on the table, because usage would require wrapping FoundationDB in a proxy or middleware of some kind to synthesize events, which can be extremely difficult to get right (due to race conditions, atomicity issues, etc). Without events, apps can find themselves polling or rolling their own pub&#x2F;sub metaphor over and over again. If anyone with sway is reading this, events are very high on the priority list for me thanx!

Seems like a pretty comprehensive release in terms of OS and language support.One oddity I see is that the ruby gem is not available on rubygems.org and therefore cannot be easily installed and maintained using the ruby package manager which is a bit of a pain.

&gt; Alternatively, your application or layer can use the &quot;versionstamp&quot; atomic operations to write its own ordered log of what it is doing, or other indexing tricks.I&#x27;m not sure I understand.Are you suggesting having a second key space at `ops&#x2F;{VERSIONSTAMP}` or something where values contain enough information about the operation to be able to process changes in an indexer? The indexer could then clean up after itself, deleting the operations once they had been ingested? ... Effectively using a portion of the keyspace as a queue?

Yes. I don&#x27;t know how well documented it is, but there is an API (well, system keyspace) that can configure the database to log transactions for a selected key range (up to and including the whole database) into another selected key range. It is used by backup and asynchronous replication tools. The format of the configuration keys and transaction logs should be considered less stable than the core key&#x2F;value API, which basically never breaks backward compatibility, so by using it you are shouldering a maintenance burden to keep up with e.g. changes in the log format and new types of mutations when new versions of FoundationDB come along. So it shouldn&#x27;t be used too casually. But it&#x27;s there.Alternatively, your application or layer can use the &quot;versionstamp&quot; atomic operations to write its own ordered log of what it is doing, or other indexing tricks. Depending on your data model this might be able to be much more efficient. For example, for external indexing you probably don&#x27;t need to preserve a history of prior values but only be able to identify all the values that have changed. This can be done with a very simple and compact index that doesn&#x27;t need to duplicate all the data to be indexed.

This is wonderful news. I built a proof-of-concept realtime collaborative editor on top of foundationdb a few years ago, and was very disappointed when I couldn&#x27;t use it in production. I&#x27;m really excited to use this in some projects I&#x27;m working on.Quick question: I know there&#x27;s a watch API, but is there any way to subscribe to a change feed from foundationdb? I&#x27;d like to consume the FDB event log to do external indexing &amp; map-reduce work.

Apple open-sources FoundationDB