Repository navigation
Conversation
|
Oh sorry. I forgot to add. The tests don't pass because test cases would have to be updated to take into account the fact that IPs are buffered. By the way, when I said backward-compatible, I meant the interface is backward-compatible. If a component currently expects IPs to be sent immediately without buffer, it'll break. So I don't know rather it's good to enforce some best practices like always disconnect after sending and always expect IPs to arrive as one connection rather than "split up" by types. This is why I PR'ed for feedbacks. ;) Thanks! |
|
If we're buffering IPs, then the important thing to take care of is also that they're sent out in the same order as they've been received. This would simplify a lot of flows. For example, if I have a component that fetches the contents of a web page asynchronously, and I send it a bunch of URLs, right now the results are sent out in whatever order the async callbacks fired. Instead, they should be sent out in the order the component received the original URLs. |
|
Hmmm. That's true. Do you think this is the only major con to it? I used to have this as a I haven't had a chance to play around with |
|
A bit of continuation to this, from the FBP book:
We should ensure that NoFlo obeys these rules, also with async components. |
|
...and even more from the FBP mailing list:
|
|
It's been a while! Yes, the description of an FBP system makes a lot of sense here. Let me confirm I have a good grip on this before moving on to fix this. Do you think the idea of buffering violates any of the constraints?
The problem may lie in that manipulation may take place after a packet has been sent as you can access the buffer after sending packets. Would that be one of your concerns? If that's the case, would it be a good idea to make this into a separate component built on top of I obviously wrote it to suit my needs so there are a lot of blindspots for me. Please feel free to point them out as I want to make this work for the NoFlo community. |
|
Thinking more on this, I just want to blast this out to see if my thinking is logical. The problem I'm trying to solve here is that FBP documents are sometimes hierarchical. That is, each sub-group is not necessarily the only child of the parent group. Therefore, under the way current components are constructed, as soon as you don't send an IP out after manipulating it upon receiving it, the paradigm breaks. This is because this assumes the grouping structure to be linear (i.e. each sub-group is the only child of its parent). There's quite a few examples of existing components that follow this pattern. One is the Throttle component. It pushes incoming IPs into a queue and release upon If hierarchical grouping is allowed for the end users, the biggest issue then is that each receiving function (i.e. This is obviously not a problem if you record each incoming IP and act on the current IP based on the IPs collected so far. The more likely case in real life is that you have to manipulate the IPs en masse within a particular group. Another example would be to turn all IPs in a specific grouping (possibly identified by some regex on the group) into an array. Now, one may argue that this should be implemented as a graph. A process extracts the matching group and forwards only that part to a process that combines the IPs into an array. Then yet another process puts the array (now just one IP) back into the spot from which the original IPs were extracted. I'm completely okay with this approach and I agree that although this greatly complicates things, it does align with the spirit of FBP. And of course you can reuse a generalized version of the above as a graph. I'm looking more for a standardized way to do something like this because if the graph approach is acceptable and preferred, we should not encourage component authors to write a component that processes any connection that postpones processing with a hierarchical grouping structure. If this is not the desirable approach, then the solution is what I was trying to accomplish, which is to provide a Port that allows programmers to preserve the hierarchy while manipulate the IPs before sending them out. Though I must say I'm having second thoughts on this as I think the graph approach may be better. Is there anything that I'm missing? I feel like this could be brought over to the FBP Google Group. I'm afraid that this is too contextual or too NoFlo-specific. Do you think it would be appropriate? |
|
I realized that there is a better way to implement this functionality as an external package. I have it implemented and just testing it internally. Will update to this thread once it's good to go. Keeping NoFlo's core small is a better route now that I see how it should work. Closing this but will post updates on that package. |
|
Thanks for the work on this! Let me know how that external package proceeds :-) |
Update yargs to version 4.3.2 🚀
Phew. This is a big PR. So here we go:
This may be familiar to you:
This is quite common in the existing libraries and I find myself doing this over and over again. Not only that, when I need to manipulate the data in batch, like examining the groups and deciding to send the data forward or not, I need to do something like this:
Doing either the first example or the second example repeatedly is not fun, nor is it correct: you lose the structure of groupings. Instead of having something like an XML-like tree structure, which groups are supposed to provide, by the time you access it, it's linear.
Hence this PR. It expands the role of the port to allow each port to have a buffer of incoming/outgoing connection. It flushes itself every time a connection is disconnected. The primary benefit is that until you disconnect, you know that you still have the data at bay and you can manipulate it. So for example:
This is a trivial example but you can see that I can choose to manipulate all the "sent" groups/data before they actually get sent. Great for batch operations.
Here are the side benefits:
beginGroupbefore they get adata. This is not how the mental model of FBP should work (at least not mine :p)Now this was done a while back. I have been testing it in production so it was before the style change and me knowing that there was a test directory. I will throw in tests for this but I need to know if this is something from which noflo can benefit. I imagined it could be a separate
class BufferedPort extends Portbut I think it's not necessary because it's backward-compatible.Let me know if I can clarify any aspect of this since this is a pretty big change in the codebase.