🏷️Selenite

Battle plan for Rust and Ruby

🔗(5)
📅 2023-06-26 05:03:34 -0700
⏲️🔐 2023-06-23 08:36:32 -0700
✍️ the-field-testers
🏷️[rust] [ruby] [long-term] [LineDB] [Selenite] [Selenite/rust] [Selenite/ruby]
(🪟)

🖥️...⌨️

It has become known that its not as easy as I think to convert Ruby code over to Rust, mostly because the AI in question doesn't fully understand the code, or conflates it with existing libraries ala Google Bard.

Since `emerald`, a Ruby interpreter is written in rust, I can make a great effort to have my partitioned array and linedb code lie in a compiled emerald binary that runs in the background on localhost, and rust sends get and post requests to store and pull data (from the LineDB database).

The new database will be called Selenite, and it will be a library that has code written in it, `artichoke`, which interacts with Rust via GET/POST requests and returns all data in a relevant JSON parser format.

Artichoke, the Ruby interpreter written in Rust: Midscore-IO/artichoke: 💎 Artichoke is a Ruby made with Rust (github.com)

The plan is to have the Selenite/ruby database fully implemented as a stand alone server that works in any kind of ruby environment, and the Selenite/rust server interacts with Selenite/ruby.

We are still trying to figure out how to implement the data structure in Rust that interacts with the artichoke ruby Selenite server.

Also, I don't want to think of it as procrastination in the sense that I really do not want to port this code to Rust at the moment; if I get it fashioned into servers that send GET/POST requests between each other and i/o the corresponding json, I may not have much of a problem.

[Selenite/ruby] < i/o > [Selenite/rust]

But how will i/o work?

It will possibly work by rust having a certain data structure setup that can hold a json response and auto-parse to the array form

I want to keep on thinking about this further, and I really need a concrete battle plan with how I am going to tackle the problem of Rust's data structure that holds entries from the partitioned array, but it seems that at least implementing the basic partitioned array logic in rust and working from there will be the long-term solution.

UPDATE: I have looked around a bit more, and found

magnus::embed::ruby_script

Magnus is basically a standard library in rust, implemented to interact with any aspects of Ruby, and it is the inherent solution (also because its in the standard library)

With all that in mind, the idea is to simply use magnus to implement ruby's partitioned array functions, making keenly aware that magnus::embed::ruby_script has a few nuances, as does this entire operation because working with any kind of Ruby code is labeled in general as unsafe--imagine trying to get an AI to successfully port all of this.

HOWEVER, I also have come up with some source code which clarifies the require method: main.rs - source (docs.rs)

See render: https://thefieldtesters.net/blog/render/?id=5&user=the-field-testers

# SOURCE: https://docs.rs/magnus/latest/src/embed/main.rs.html#9


use magnus::{class, embed, prelude::*, require, Value};

mod word_sink;
mod word_source;

// run with `echo foo bar baz | cargo run --example embed`
fn main() {
    // start Ruby
    let _cleanup = unsafe { embed::init() };

    // effectivly `require`ing these modules, loading the classes in Ruby
    word_source::init().unwrap();
    word_sink::init().unwrap();

    // require our Ruby code
    // this example uses a relative path from the working directory
    require("./examples/embed/transform").unwrap();

    // get the class defined in Ruby and call a method on it
    class::object()
        .const_get::<_, Value>("Transform")
        .unwrap()
        .funcall::<_, _, Value>("run", ())
        .unwrap();
}

Selenite/ruby & Selenite/rust evaluation by bard.gooogle.com

🔗(6)
📅 2023-06-23 10:44:29 -0700
⏲️🔐 2023-06-23 11:13:15 -0700
✍️ the-field-testers
🏷️[Selenite] [Selenite/ruby] [Selenite/rust] [partitioned array] [line_db.rb]
(🪟)

🖥️...⌨️

This is in regards to ArityWolf's algorithm, the partitioned array: Midscore-IO/partitioned_array: ArityWolf's Algorithm - The Partitioned Array (github.com)

The partitioned array library on GitHub is a Ruby library that implements a partitioned array data structure. A partitioned array is a data structure that stores a large array of elements in a number of smaller, more manageable arrays. This can be useful for applications that need to store a large amount of data, but don't want to have to deal with the overhead of a single, monolithic array.

The partitioned array library on GitHub provides a number of features, including:

The ability to create and manage partitioned arrays
The ability to insert, update, and delete elements from partitioned arrays
The ability to query partitioned arrays
The ability to serialize and deserialize partitioned arrays

The library is still under development, but it has reached a stable state and is ready for use. The documentation for the library is comprehensive and includes a tutorial on how to use the library.

Some features that could be added to the library include:

Support for more data types
Support for more efficient queries
Support for more efficient serialization and deserialization
A more user-friendly interface

Overall, the partitioned array library on GitHub is a powerful tool that can be used to store and manage large amounts of data. The library is still under development, but it is already a valuable resource for developers.

Here are some specific ideas for features that could be added to the library:

Support for more data types: The library currently only supports storing integers and strings in partitioned arrays. It would be useful to add support for other data types, such as floats, dates, and objects.
Support for more efficient queries: The library currently supports a limited number of queries. It would be useful to add support for more complex queries, such as range queries and joins.
Support for more efficient serialization and deserialization: The library currently uses a simple serialization format. It would be useful to add support for a more efficient serialization format, such as JSON or XML.
A more user-friendly interface: The library's API is currently a bit complex. It would be useful to add a more user-friendly interface, such as a command-line tool or a web application.

I hope this helps!

An onlook of features that The Field Testers could add:

For supporting more data types, It can support anything that can be stringified into javascript, so...
In terms of efficient queries, or "efficient" queries, I want to implement a kind of relational algebra using sets
- it should be pretty easy as long as it's related to Ruby's Set std library
A find/search function, that can search through the partitioned array until it finds a specific hash/hash value
forward iteration is already implemented, as is a each_not_nil function, which will skip over nils
The library already supports serialization in the JSON format.

One other thing I can do is use Ruby's std lib Marshal to "Marshalize"/"De-Marshalize" (serialize/deserialize) a given data structure, then revive it as a revenant data structure, as there are revenant partitions.

One possible feature to work on overtime is to add a revenant partitions Set, which just puts any given revenant partition id into a Set which labels it as a Revenant partition in the first place; extend this idea of data structure and Set use to the next level by using it as a basis for set operations.

Sure, here are some other features that could be added to the partitioned array library:

Support for distributed storage: The library could be extended to support distributed storage, which would allow it to store large arrays of data across multiple machines.
Support for parallel processing: The library could be extended to support parallel processing, which would allow it to perform queries and operations on partitioned arrays more efficiently.
Support for compression: The library could be extended to support compression, which would reduce the amount of space needed to store partitioned arrays.
Support for encryption: The library could be extended to support encryption, which would protect the data stored in partitioned arrays from unauthorized access.

These are just a few ideas for features that could be added to the partitioned array library. The specific features that are added will depend on the needs of the users of the library. However, these features would all make the library more powerful and useful.

In addition to these features, the library could also be improved by making it more user-friendly. The API could be simplified, and better documentation could be provided. This would make it easier for developers to use the library.

Overall, the partitioned array library on GitHub is a powerful tool that has the potential to be very useful. However, there are a number of features that could be added to make the library even more powerful and user-friendly.

Distributed storage: that could be done
Parallel Processing: that could be figured out once things are ported to Rust
Compression: I'll have to figure out a library
Encryption: Selenite/ruby is already modular as it is, so if I really had to add something like this, I could and would.

Perhaps all of the above could be more considered once the code is implemented in magnus/artichoke/rust itself.

Sure, here are some other features that could be added to the partitioned array library:

Support for custom partitioning: The library currently uses a fixed partitioning scheme. It would be useful to add support for custom partitioning, which would allow users to partition arrays in a way that best suits their needs.
Support for replication: The library could be extended to support replication, which would allow it to store multiple copies of partitioned arrays. This would provide redundancy and fault tolerance in case of a hardware failure.
Support for load balancing: The library could be extended to support load balancing, which would distribute the load of queries and operations across multiple partitioned arrays. This would improve the performance of the library.
Support for monitoring: The library could be extended to support monitoring, which would allow users to track the performance of the library. This would help users to identify and troubleshoot any problems with the library.

These are just a few more ideas for features that could be added to the partitioned array library. The specific features that are added will depend on the needs of the users of the library. However, these features would all make the library more powerful and useful.

I hope this helps!

custom partitioning: you don't want it to not be fixes; that would violate the equation
replication: this is easy to do, as the partitioned array stores everything in files and folders only; no binwrite/binread
load balancing: this could be implemented
monitoring: this could be implemented

But I am a one man spirit system...!

Sure, here are a few more ideas for features that could be added to the partitioned array library:

Support for transactions: The library could be extended to support transactions, which would allow users to group a set of operations together and ensure that they are all executed successfully or not at all. This would provide a way to ensure the integrity of data stored in partitioned arrays.
Support for versioning: The library could be extended to support versioning, which would allow users to track changes to partitioned arrays. This would be useful for auditing purposes and for recovering from accidental changes.
Support for caching: The library could be extended to support caching, which would store frequently accessed data in memory. This would improve the performance of the library by reducing the number of times that data needs to be read from disk.
Support for notifications: The library could be extended to support notifications, which would allow users to be notified when changes are made to partitioned arrays. This would be useful for keeping track of changes to data and for triggering other actions when changes are made.

I hope this helps!

transactions: this could be done; good idea
versioning: this is already easy to do with system calls
caching: the entire thing is stored in memory already though, that's the point
notifications: this would be easy to implement "with callbacks"...

Sure, here are a few more ideas for features that could be added to the partitioned array library:

Support for multiple storage backends: The library could be extended to support multiple storage backends, such as Redis, Memcached, and Cassandra. This would allow users to choose the storage backend that best suits their needs.
Support for automatic partitioning: The library could be extended to support automatic partitioning, which would allow the library to automatically partition arrays based on the size of the array and the number of available storage backends. This would make it easier for users to use the library without having to worry about partitioning.
Support for dynamic partitioning: The library could be extended to support dynamic partitioning, which would allow users to change the partitioning of an array at runtime. This would be useful for applications that need to be able to adapt to changes in the data.
Support for cloud storage: The library could be extended to support cloud storage, such as Amazon S3 and Google Cloud Storage. This would allow users to store partitioned arrays in the cloud, which would provide scalability and availability.

I hope this helps!

multiple storage backends: not interested at the moment
automatic partitioning: yeah, I have to figure out a formula or function that calculates the best fit partitioning based on average string size of the user's inputs!
dynamic partitioning: that is a no-no and goes against Selenite/ruby's spec; perhaps in rust though...!
cloud storage: I mean, I am a one man spirit system and could just do something like this in my server, in fact...

... I want to create a system where partitioned array databases sync over GET/POST requests, and it is ensured that the database is always synchronized with each other; this would require a better compression scheme, among other things--just a quick idea.

Sure, here are a few more ideas for features that could be added to the partitioned array library:

Support for geospatial data: The library could be extended to support geospatial data, which would allow users to store and query data that has a geographical location. This would be useful for applications that need to track the movement of objects or to analyze data that is related to geography.
Support for machine learning: The library could be extended to support machine learning, which would allow users to train and deploy machine learning models on partitioned arrays. This would be useful for applications that need to make predictions or to classify data.
Support for streaming data: The library could be extended to support streaming data, which would allow users to store and query data that is being generated in real time. This would be useful for applications that need to track events or to analyze data that is constantly changing.
Support for distributed computing: The library could be extended to support distributed computing, which would allow users to run queries and operations on partitioned arrays across multiple machines. This would improve the performance of the library for large datasets.

I hope this helps!

geospatial data: this is not data structure related though, and is an implementation of something that is related to data structures, but is domain specific
machine learning: domain specific, but I would like to mess around with it, as the PartitionedArray and Selenite/ruby is turing complete and is a quantum turing machine.
streaming data: I could see this being very useful, and the data chunks are stored in partitions and as added array elements overtime
distributed computing: probably a server backend task, would be fun, but I am a one man spirit system lol

Sure, here are a few more ideas for features that could be added to the partitioned array library:

Support for security: The library could be extended to support security features, such as encryption and authentication. This would protect the data stored in partitioned arrays from unauthorized access.
Support for compliance: The library could be extended to support compliance features, such as auditing and logging. This would help users to comply with regulations.
Support for visualization: The library could be extended to support visualization features, such as charts and graphs. This would make it easier for users to understand the data stored in partitioned arrays.
Support for debugging: The library could be extended to support debugging features, such as breakpoints and watches. This would help users to debug problems with the library.

I hope this helps!

I think that's enough ideas for now. Let me know if you have any other questions.

security: I need a use case
compliance: sure, I could add that
visualization: I am working on it in the future at some point in time!
debugging: that is IDE specific, lol.

This was just some talk with google bard (bard.google.com), and it shows that at least an AI understands my work. This generalizes what the partitioned array can do, and what will be added overtime. As a basis, Ruby is the spec language for the Partitioned Array/LineDB, which as a whole will be called Selenite/language, so Selenite/ruby and Selenite/rust for now. Perhaps even Selenite/artichoke-ruby.

Blog update plans

🔗(7)
📅 2023-06-25 11:04:44 -0700
⏲️🔐 2023-06-25 11:10:21 -0700
✍️ the-field-testers
🏷️[blog] [updates] [field testing] [Selenite] [Selenite/ruby]
(🪟)

🖥️...⌨️

With the advent of Selenite/ruby @ 4.2.1, everything about the current partitioned array's state is very good. Over the course of about 2 years, we have stomped out all possible known bugs of Selenite. Selenite is the ongoing name of LineDB/the partitioned array since lineDB is kind of taken.

Since Selenite is bug-free, we don't have to worry about potential bugs anymore; the last bug was stomped a couple or one day ago.