Category: xst

I'm not sure I'm done with this XST experiment, but I've come to some conclusions that I want to set down. Having done that, maybe I'll know what to do next.

Mistakes? Technical Debt? Better ideas? All the above?

OK with that out of my system, what shall we code today? It goes so nicely, until oops.

I'm not out of ideas, but I am out of directions, and my momentum is low. What shall I do?

The swirl that is my life leads me to think about duplicates and such. Very speculative musings, no conclusion. Sort of Captain's Log Stardate 77691.1.

The statistics code is much better now. But a little widget might be just the thing. Turns into a medium-sized widget, but I think it's an improvement.

The statistics code runs, but is written in 1960's style—yes, I DO know what 1960's code looks like—and needs improvement. An interesting series of events with a fine result.

Commonly, when we group data by some criteria, we desire summary information about the groups: counts, totals, averages, and so on. We take a couple of nice steps before my brain is fried.

Yesterday's experiment with `group_by` seemed very successful. What does it tell us about what we're doing and what we should be doing? Bit of a retrospective.

Let's try an experiment, along the lines of my sketch at the end of this morning's article. It goes very well. Very well indeed!

I'm here for another step on the way to the grouping operator. I have at least one new realization going in. More than one coming out. Some code progress as well.

Time to push a bit further on the grouping operations. I don't get far before realizing that I need to break away and try again later. Not great, but good.

A bit of research into the past of Extended Set Theory suggests an interesting possibility for the future, and provides the joy of working something out on my own. Includes some possibly amusing history about how primitive man created such documents.

I'm continuing directly on from the preceding article, to create my XGroup implementation. It may turn out that a break was needed. We'll see.

This morning I'm working toward a join operator, by way of a grouping operator. I have vague ideas, and need some help from my code.

Fancy as my new streaming select is, I think it's, well, not quite the thing. Easily replaced, but do I need to pivot? I surprise myself.

OK, new rule, we're going to read XFlatFile sets into memory. Should we refactor to this new result, or TDD a new class? Let's refactor. Works quite well!

It's 0415 and I am thinking about buffering. I create an XSet with a billion records in it. Python does not sneeze.

The thing about learning is that when we have sucked most of the learning juice out of some fruitful idea, there is often work to be done to get the thing finished. We have at least two of those hanging fruits right now. One of them has an interesting effect!

The Powers That Be have invited me to discuss the risks around this Python XST effort, on the assumption that it is anything other than play.

There's no doubt that we can create sets or set operations that stream their results, avoiding creating large temporary sets in memory. There are issues to think about. First experiment works!

In which, your intrepid author considers times now and times then, assessing whether and how to adjust what we do with XST given the new reality.

A check of our list of ideas leads to discoveries, to key decisions, and to reflection on the realities of software development. Quite a span, as perhaps it always should be. Interesting article reference included.

Ran across this. Rather nice. Then I took one more step. Off a cliff. Fortunately, I didn't look down.

We've been working on calculated fields. Let's see about getting them built into sets. Yucch! I touched a debugger.

Our work on calculations raises some longer-term concerns about large dataset and statistics. Are we in big trouble?

Let's fetch values from XST records for our Expressions. I expect this to go smoothly. It does. We discuss small steps. No, smaller than that.

I have a bit of time, with distractions. Let's see about error conditions in our expressions.

We need to work on assignments and field values. I was thinking values, and then we do assignments instead. That's where the path looked best.

We'll take some steps along the expression path. I'm slightly questioning part of what we have. The path is zig-zag but leads to a good place.

Spikes for the expression parsing have taught us enough. Let's see how we can best move from the spike code into decent production objects. There's one somewhat large issue: symbols. No, two: data types.

Back to the interpreter. Shall we try the lambda thing again, or something else? Lambda seems fraught. Partial function FTW.

We can parse a simple expression. Let's sketch the expression interpreter. Bear gives us a nip, but we have progress.

I think I'll break down and do expressions. I'm thinking a simple Dijkstra-style parser will do the job for us. Let's see if we can find a simple way to develop one.

Suppose we want a set where one element is the sum of two others. What about calculating a scope? I am interrupted and then spot a squirrel.

Somewhat enlightened about how early XSP systems worked, there are a few ways we might go. I need too reflect on possibilities and pick a direction.

Study of some ancient scrolls leaves me thoughtful, and a bit disappointed.

You probably do not want to read this. I'm writing down my thoughts, in the vain hope of figuring out what they are and what they should be. Dave Childs was kind enough to send me some links to articles and other materials about XST. I include them at the end of this article, for reasons. The materials gave me some things to think about.

I was whining earlier about the difficulty of writing tests that assert about elements inside a result set. A thought has come to me.

I want to work on some indexing operations. Along the way I rediscover XFlatFile's existing scope set feature. Just what we need. Or is it? Includes a judicious rollback.

Laurent challenged me to find a better, more elegant formulation for getting the correct re_scoping set from a provided renaming set. I thought I had it. Then I thought I hadn't. Then I thought I had.

We have a long-form cardinality method. Let's use the len function and require it as part of the implementations.

Let's look around and see what we might do. I'll even make a list. Jira is an abomination upon the land.

I have received a bug report! This is great news! Someone is paying attention!

Putting a specialized rename into our flat set showed us that the symbol table there is quite ad hoc. Let's make it more like a set. (Turns out: no.)

Since the Flat sets know their field names once and for all, we could use a better rename that doesn't copy the data. Simple enough, rather nice.

If there was a kind of set that was expressed as a function, we could possibly pipeline operations, reducing memory impact. Is that possible, and is it a good idea? So far, maybe not.

What if a function IS a set instead of returning a set? This might be significant.

We try and succeed in implementing an XFlatFile that refers only to a subset of the file. This is a big deal!

I'm working on the idea of an XFlatFile that only reads a defined subset of the file. Am I working without a net? Not really.

Plus wasn't the right operator for union. Let's update that and then a few other operators.

I had thought that I'd work on the flat data. My thoughts led me astray, to a glimmering of a possibly good idea. So I'll just code something cute to finish the morning's work.

Thinking about rename in the Flat implementation leads to discovery of an interesting defect, and some thinking about use of the powerful generality of set theory.

In an astounding flurry, we are going to build a rename operation.

Childs has defined two "re-scope" operations. Let's see if we can implement one of them, and if we're glad we did.

We'll start with a simple removal of the requirement for implementations to implement `__contains__`. After that, we'll see. (And that's not what happens.)

In your absence, I made a simple but significant change. I had an interesting idea. And an important observation. And more.

We make some progress on our flat files, but progress is slow and there are a lot of words here. Best skim or skip?

Let's see about getting set creation sorted. We need to be better able to create sets with any possible base implementation. Deep confusion ensues.

To process flat files, we want to avoid leaving a file open, and we don't want to open it a zillion times. Do we have to invent buffering? Perhaps not.

Our experimental tests one flat records look good. Let me report on my off-line work, and then let's see what's next.

Let's get started on a flat-file-focused form of set. We make a bit of progress.

I want to begin by thinking about storage: how do we best produce specific memory / file formats? The article turns out to be pure speculation, but perhaps useful speculation.

Having prepared better than we did yesterday morning, let's proceed with wrapping our `frozenset` with a class of our own.

Let's have another try at wrapping our `frozenset` in an object that will work for us. Things go much better ... so far.

I make some useful initial observations, and then, well, I crash and burn. And mix a metaphor.

I think that this morning, I'll take a small step toward having more than one implementation of a set's data.

If I'm going to get serious about this Extended Set Theory thing, a tiny bit of design thinking seems to be in order.

It is 0253 hours. I have a report and another report.

Lets do `project`, as in projection, as preparation for trying a generator approach to set expressions. This part should be easy.

There's a thing I want to do with this XSet stuff, and I don't quite know how to do it. I need to better understand iterators and generators.

In which, we report on a nifty little thing, and then, well, I don't know yet what I'll do. Some concluding remarks on what it is that I do here.

In which, I show you what I'm dealing with, and mention an insight.

Let's see how we can select records from an XSet, using the `restrict` operator.

I propose to push a bit further on the use of Python frozenset to do a little Extended Set Theory. I mention symmetric difference!

Some random reading, and the messing about that I've done in the FAFO series, leads me to want to explore Extended Set Theory in Python. I do not expect it to be useful but it might be interesting.

Yesterday's search for strawberries in the XST patch discovered problems. I want to at least double check what we did. TL;DR: It all works. Odd morning.

I'm pressing forward with the Lispy thing, but I have some concerns. I find it difficult to think about but I think I have an angle. (Spoiler: It's a wrap!)

Just a bit more playing with the Lispy Calculator to while away a few minutes in the afternoon.

Thoughts and observations. Stuff and nonsense.

I'm going to push forward with this LISP / Scheme dialect. I'll begin by explaining why, and why not.

It's still the weekend, so I'm going to follow Peter Norvig's Python LISP Implementation a while and see where it takes me.

It's my weekend and I'll try if I want to. You could try too if it happened to you. Spoiler: This takes a very weird turn. Final line: Your move, Bill!

I guess there's nothing for it but to figure out how to rewrite, i.e. refactor, a set operation based on the existence of helper structures. But how? I have ideas but are any of them any good?

Bill Wake is trying to get me to think in terms of trees. I don't want to, but he does have some good ideas. Thanks, Bill!

Today I plan to experiment with creating some form of expressions that might be optimized. I expect to stumble a lot. Come along, point, and laugh.

No, not smiles and frowns. Algebra. At this moment I don't think much code will be done today. Feel cute, might delete later.

It's 6 AM and I have an idea. This could be very good or very bad.

I don't love the interface for adding functions into an XSet. And I want to add them 'one level in'. Will hilarity ensue? Probably not, but something will happen.

Some HNY thoughts, and more on the function as an element idea. Joy, philosophy, code. What's not to like?

I was thinking before I got up about median and mode. Then I had a truly marvelous idea.

I want Lua tables to be more useful as XSets. There's a hard way. But the current design also offers an easy way. (The answer will surprise you. It surprised me.)

I'm on a path to make ordinary tables behave like XSets. But first, I have to figure out how this thing actually works! Much musing, then some code.

I have in mind small things for today, starting with an interesting and confusing mistake left over from yesterday.

No, I'm not hearing voices. But the code does tell us things, just like any working material. We need to learn to listen. Today, we listen and the result is good.

A look at the code. Maybe a bit more on stats. P.S. I learn something and ditch almost all the code I wrote this morning.

It's Christmas, I'm waiting for the household to wake up, and I enjoy what I'm doing. Perfect holiday so far!

Today I plan to get grouping and summing working. Who knows, it might happen. If not, there's always tomorrow or my birthday.

Let's think about what the current drafts of summing and grouping tell us about our system. Then code (anyway).

Today, rather than make any deep progress, I plan to work on something I consider interesting, sums, averages, and grouping. I promise to publish this even if it explodes. (It doesn't, quite.)

Transformations, optimizations, and the relationship between OO and XST. Got some thinking to do. You get to watch, if you're tough enough. At least a tiny bit of code.

Step by step, inch by inch, slowly we turn long searches into more direct accesses.

I don't want to get stuck in a never-ending series of new setops: there won't be much learning there. Where's the beef?

I'm sure a lot of you have been saying 'Yes, but what about tuples?', or 'Why XST anyway?'. Today, we address those fascinating concerns.

Let's work on those new operations a bit.

Short morning today. I have a tentative plan for indexes. I'll scribble some sets. Might code.

Last night's Zoom Ensemble netted me a few ideas. I'll start exploring those today. Hilarity or perhaps something good will ensue. I can't wait to find out.

Further reading leads me to think about design, and design motivation. Castles in the air. Or underground. Good stuff happens.

In conversation with Bill Wake and with the Internet, I have an idea for something to try. And I'm just about ready to assess where we are and where we should go.

I'm not sure whether this will be useful, but Bill Wake gets the credit if it is.

Bill Wake was trying to hammer an idea into my head. I must think about that. And I have a small idea of my own.

Time to work on the actual restrict operator for CSV, since the pattern-maker experiment was a success.

There's no way around it, I've got to work on the fast restrict today. Might not finish. We'll see.

Getting started with CSV data. And reporting a conversation.

Reflection leads me to focus a bit more on set operations, and less on internal methods. Does this call for a new layer? Also: real technical debt! Updated with idea from Carl Manaster!

I'm going to try to create pipelines using coroutines. I think they may make for a more expressive interface. I turn out to be partially right.

How do you design a thing like this, Brain? Same as everything else? Or not?

I was puzzling over an issue with 'union' and gained an insight that either I've never had, or that I had lost. Whee!

I found the easy way to build an iterator in Lua, so we'll do that and see whether it improves the code as much as I think it will.

I think I'm going to start on restrict today. There are some issues around atoms.

Last night I understood how to do something with XST that I've not in the past been able to do. So let's talk about why XST is interesting and what one might do with it.

I have a random idea about the data structure for sets, so thought I'd give it a try.

Save me, I'm thinking about Extended Set Theory again.