Category: pyxst

I'm not sure I'm done with this XST experiment, but I've come to some conclusions that I want to set down. Having done that, maybe I'll know what to do next.

Mistakes? Technical Debt? Better ideas? All the above?

OK with that out of my system, what shall we code today? It goes so nicely, until oops.

I'm not out of ideas, but I am out of directions, and my momentum is low. What shall I do?

The swirl that is my life leads me to think about duplicates and such. Very speculative musings, no conclusion. Sort of Captain's Log Stardate 77691.1.

The statistics code is much better now. But a little widget might be just the thing. Turns into a medium-sized widget, but I think it's an improvement.

The statistics code runs, but is written in 1960's style—yes, I DO know what 1960's code looks like—and needs improvement. An interesting series of events with a fine result.

Commonly, when we group data by some criteria, we desire summary information about the groups: counts, totals, averages, and so on. We take a couple of nice steps before my brain is fried.

Yesterday's experiment with `group_by` seemed very successful. What does it tell us about what we're doing and what we should be doing? Bit of a retrospective.

Let's try an experiment, along the lines of my sketch at the end of this morning's article. It goes very well. Very well indeed!

I'm here for another step on the way to the grouping operator. I have at least one new realization going in. More than one coming out. Some code progress as well.

Time to push a bit further on the grouping operations. I don't get far before realizing that I need to break away and try again later. Not great, but good.

A bit of research into the past of Extended Set Theory suggests an interesting possibility for the future, and provides the joy of working something out on my own. Includes some possibly amusing history about how primitive man created such documents.

I'm continuing directly on from the preceding article, to create my XGroup implementation. It may turn out that a break was needed. We'll see.

This morning I'm working toward a join operator, by way of a grouping operator. I have vague ideas, and need some help from my code.

Fancy as my new streaming select is, I think it's, well, not quite the thing. Easily replaced, but do I need to pivot? I surprise myself.

OK, new rule, we're going to read XFlatFile sets into memory. Should we refactor to this new result, or TDD a new class? Let's refactor. Works quite well!

It's 0415 and I am thinking about buffering. I create an XSet with a billion records in it. Python does not sneeze.

The thing about learning is that when we have sucked most of the learning juice out of some fruitful idea, there is often work to be done to get the thing finished. We have at least two of those hanging fruits right now. One of them has an interesting effect!

The Powers That Be have invited me to discuss the risks around this Python XST effort, on the assumption that it is anything other than play.

There's no doubt that we can create sets or set operations that stream their results, avoiding creating large temporary sets in memory. There are issues to think about. First experiment works!

In which, your intrepid author considers times now and times then, assessing whether and how to adjust what we do with XST given the new reality.

A check of our list of ideas leads to discoveries, to key decisions, and to reflection on the realities of software development. Quite a span, as perhaps it always should be. Interesting article reference included.

Ran across this. Rather nice. Then I took one more step. Off a cliff. Fortunately, I didn't look down.

We've been working on calculated fields. Let's see about getting them built into sets. Yucch! I touched a debugger.

Our work on calculations raises some longer-term concerns about large dataset and statistics. Are we in big trouble?

Let's fetch values from XST records for our Expressions. I expect this to go smoothly. It does. We discuss small steps. No, smaller than that.

I have a bit of time, with distractions. Let's see about error conditions in our expressions.

We need to work on assignments and field values. I was thinking values, and then we do assignments instead. That's where the path looked best.

We'll take some steps along the expression path. I'm slightly questioning part of what we have. The path is zig-zag but leads to a good place.

Spikes for the expression parsing have taught us enough. Let's see how we can best move from the spike code into decent production objects. There's one somewhat large issue: symbols. No, two: data types.

Back to the interpreter. Shall we try the lambda thing again, or something else? Lambda seems fraught. Partial function FTW.

We can parse a simple expression. Let's sketch the expression interpreter. Bear gives us a nip, but we have progress.

I think I'll break down and do expressions. I'm thinking a simple Dijkstra-style parser will do the job for us. Let's see if we can find a simple way to develop one.

Suppose we want a set where one element is the sum of two others. What about calculating a scope? I am interrupted and then spot a squirrel.

Somewhat enlightened about how early XSP systems worked, there are a few ways we might go. I need too reflect on possibilities and pick a direction.

Study of some ancient scrolls leaves me thoughtful, and a bit disappointed.

You probably do not want to read this. I'm writing down my thoughts, in the vain hope of figuring out what they are and what they should be. Dave Childs was kind enough to send me some links to articles and other materials about XST. I include them at the end of this article, for reasons. The materials gave me some things to think about.

I was whining earlier about the difficulty of writing tests that assert about elements inside a result set. A thought has come to me.

I want to work on some indexing operations. Along the way I rediscover XFlatFile's existing scope set feature. Just what we need. Or is it? Includes a judicious rollback.

Laurent challenged me to find a better, more elegant formulation for getting the correct re_scoping set from a provided renaming set. I thought I had it. Then I thought I hadn't. Then I thought I had.

We have a long-form cardinality method. Let's use the len function and require it as part of the implementations.

Let's look around and see what we might do. I'll even make a list. Jira is an abomination upon the land.

I have received a bug report! This is great news! Someone is paying attention!

Putting a specialized rename into our flat set showed us that the symbol table there is quite ad hoc. Let's make it more like a set. (Turns out: no.)

Since the Flat sets know their field names once and for all, we could use a better rename that doesn't copy the data. Simple enough, rather nice.

If there was a kind of set that was expressed as a function, we could possibly pipeline operations, reducing memory impact. Is that possible, and is it a good idea? So far, maybe not.

What if a function IS a set instead of returning a set? This might be significant.

We try and succeed in implementing an XFlatFile that refers only to a subset of the file. This is a big deal!

I'm working on the idea of an XFlatFile that only reads a defined subset of the file. Am I working without a net? Not really.

Plus wasn't the right operator for union. Let's update that and then a few other operators.

I had thought that I'd work on the flat data. My thoughts led me astray, to a glimmering of a possibly good idea. So I'll just code something cute to finish the morning's work.

Thinking about rename in the Flat implementation leads to discovery of an interesting defect, and some thinking about use of the powerful generality of set theory.

In an astounding flurry, we are going to build a rename operation.

Childs has defined two "re-scope" operations. Let's see if we can implement one of them, and if we're glad we did.

We'll start with a simple removal of the requirement for implementations to implement `__contains__`. After that, we'll see. (And that's not what happens.)

In your absence, I made a simple but significant change. I had an interesting idea. And an important observation. And more.

We make some progress on our flat files, but progress is slow and there are a lot of words here. Best skim or skip?

Let's see about getting set creation sorted. We need to be better able to create sets with any possible base implementation. Deep confusion ensues.

To process flat files, we want to avoid leaving a file open, and we don't want to open it a zillion times. Do we have to invent buffering? Perhaps not.

Our experimental tests one flat records look good. Let me report on my off-line work, and then let's see what's next.

Let's get started on a flat-file-focused form of set. We make a bit of progress.

I want to begin by thinking about storage: how do we best produce specific memory / file formats? The article turns out to be pure speculation, but perhaps useful speculation.

Having prepared better than we did yesterday morning, let's proceed with wrapping our `frozenset` with a class of our own.

Let's have another try at wrapping our `frozenset` in an object that will work for us. Things go much better ... so far.

I make some useful initial observations, and then, well, I crash and burn. And mix a metaphor.

I think that this morning, I'll take a small step toward having more than one implementation of a set's data.

If I'm going to get serious about this Extended Set Theory thing, a tiny bit of design thinking seems to be in order.

It is 0253 hours. I have a report and another report.

Lets do `project`, as in projection, as preparation for trying a generator approach to set expressions. This part should be easy.

There's a thing I want to do with this XSet stuff, and I don't quite know how to do it. I need to better understand iterators and generators.

In which, we report on a nifty little thing, and then, well, I don't know yet what I'll do. Some concluding remarks on what it is that I do here.