Skip to content

Change component data wrappers to be data classes - #4

Merged
jakub-toptal merged 2 commits into
frequenz-floss:v0.x.xfrom
jkozlowicz:component-data-refactor
Oct 14, 2022
Merged

jakub-toptal merged 2 commits into
frequenz-floss:v0.x.xfrom
jkozlowicz:component-data-refactor

Conversation

@jkozlowicz

Copy link
Copy Markdown
Contributor

No description provided.

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

I didn't go in detail through the PR yet, but my first impression is we really need to start specifying units. Is SoC in percentage? Is it in Wh, kWh? Etc.

I opened a discussion so we don't derail about that in this PR: #6

@jkozlowicz

Copy link
Copy Markdown
Contributor Author

Makes sense. Units are described in the protobuf messages, e.g. for soc:

class Data(google.protobuf.message.Message):
    """Battery data."""

    DESCRIPTOR: google.protobuf.descriptor.Descriptor

    DC_FIELD_NUMBER: builtins.int
    SOC_FIELD_NUMBER: builtins.int
    TEMPERATURE_FIELD_NUMBER: builtins.int
    HUMIDITY_FIELD_NUMBER: builtins.int
    @property
    def dc(self) -> frequenz.api.microgrid.common_pb2.DC:
        """DC electricity metrics."""
    @property
    def soc(self) -> frequenz.api.microgrid.common_pb2.MetricAggregation:
        """Battery's overall SoC.
        In percent (%).
        """

Shall I copy these over from the protobuf message docstrings?

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

Shall I copy these over from the protobuf message docstrings?

Yes, please! Until we decide how to represent units better I think is a good idea to have it at least in the documentation.

@sahas-subramanian-frequenz

Copy link
Copy Markdown
Contributor

It looks like the commits are not cryptographically signed. Maybe because it was rebased from the UI?

I wonder if we can get the DCO check to verify that the commits are signed, and not just that they have the "Signed-Off" line in the commit messages.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only made a comment on the first occurrence, but when/if fixed it should be fixed on all classes. (like add units to the documentation for all attributes, moving the component_id and timestamp documentation only to ComponentData, etc.)

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

I wonder if we can get the DCO check to verify that the commits are signed, and not just that they have the "Signed-Off" line in the commit messages.

The DCO bot is provided as is, but merging will be blocked if commits are not cryptographically signed, so having it on a check is not necessary (and it will be redundant in a way)

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

@jakub-toptal BTW you can see the protobuf comments with less noise in the .proto files

@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from fe9a410 to 4df1d4d Compare October 4, 2022 13:15
@jkozlowicz
jkozlowicz requested a review from a team as a code owner October 4, 2022 13:15
@github-actions github-actions Bot added the part:tests Affects the unit, integration and performance (benchmarks) tests label Oct 4, 2022
@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from 4df1d4d to e13f52a Compare October 4, 2022 13:38
@sahas-subramanian-frequenz

Copy link
Copy Markdown
Contributor

Possible field descriptions:

battery

soc_lower_bound: the SoC below which discharge commands will be blocked by the system.
soc_upper_bound: the SoC above which charge commands will be blocked by the system.
power_lower_bound: the maximum discharge power, in Watts, represented in the passive sign convention. This will be a negative number, or zero if no discharging is possible.
power_upper_bound: the maximum charge power, in Watts, represented in the passive sign convention. This will be a positive number, or zero if no charging is possbile.

inverter

active_power_lower_bound: the maximum discharge power, in Watts, represented in the passive sign convention. This will be a negative number, or zero if no discharging is possible.
active_power_upper_bound: the maximum charge power, in Watts, represented in the passive sign convention. This will be a positive number, or zero if no charging is possible.

everywhere

active_power: the 3-phase active power, in Watts, represented in the passive sign convention.

@jkozlowicz jkozlowicz mentioned this pull request Oct 4, 2022
@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from e13f52a to 5e192cb Compare October 4, 2022 15:44
@leandro-lucarella-frequenz leandro-lucarella-frequenz added part:data-pipeline Affects the data pipeline type:enhancement New feature or enhancement visitble to users labels Oct 5, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also minor, but I would add a (W) for Watt too (as in in Watt (W)), like with Volt and Ampere, just for consistency.

@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from 5e192cb to 6749c09 Compare October 5, 2022 10:57
@github-actions github-actions Bot removed the part:data-pipeline Affects the data pipeline label Oct 5, 2022
@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from 6749c09 to 33dea98 Compare October 6, 2022 12:09
@leandro-lucarella-frequenz leandro-lucarella-frequenz added the status:blocked Other issues must be resolved before this can be worked on label Oct 7, 2022
@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

Marking this as blocked because until we have a decision on how the pipeline should work, we can't really decide on this one.

@jakub-toptal

Copy link
Copy Markdown
Contributor

Marking this as blocked because until we have a decision on how the pipeline should work, we can't really decide on this one.

This doesn't really affect the data pipeline. It doesn't change anything, except for making the wrapper classes for ComponentData dataclasses instead of normal classes with properties. It makes it more convenient to work with these.

If for some reason we decide that the representation needs to change in the context of the data pipeline, we can just change dataclass-based classes instead of property-based classes.

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

Well, is adding more processing each time you receive a message, because now you need to construct a dataclass for each grpc message. So I agree is nicer to work with in general but if we don't expect users to work with them directly, there is little value in adding the extra processing.

And it does change something, how these objects are constructed, so it is even a breaking change (although I'm not sure if anybody else is using it).

It is IMHO an unnecessary change if these classes won't be omnipresent in the data pipeline or higher levels, this is why I think we should keep it on hold for now.

@leandro-lucarella-frequenz

Copy link
Copy Markdown
Contributor

The documentation improvements are quite nice though, that might also lean the balance towards merging it anyways :D

@ela-kotulska-frequenz @sahas-subramanian-frequenz any opinions on this? Do you know if these classes are constructed somewhere else outside of the SDK (so if the breaking change has any impact)?

@jakub-toptal

Copy link
Copy Markdown
Contributor

@ela-kotulska-frequenz @sahas-subramanian-frequenz please let us know what you think. If this is not a breaking change for our current SDK users, I'd go for it, because it makes it easier to work with, especially in tests. New classes don't need to be provided a protobuf message with raw data. Look at this example:

https://github.com/frequenz-floss/frequenz-sdk-python/blob/v0.x.x/tests/utils/data_generation.py#L28-L81

we have to create a quite complex protobuf message to get an instance of BatteryData. With the updated classes we could simply use something along these lines:

BatteryData(
    component_id=1,
    timestamp=ts,
    soc=1.0,
    soc_lower_bound=-5.0,
    soc_upper_bound=5.0,
    capacity=10.0,
    power_lower_bound=-20.0,
    power_upper_bound=20.0,
)

@thomas-nicolai-frequenz

Copy link
Copy Markdown

If this is not a breaking change for our current SDK users

We can always break things! Please don't take existing code into account. Its not important.

@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from ff70fb4 to 005f85b Compare October 12, 2022 12:10
@sahas-subramanian-frequenz

Copy link
Copy Markdown
Contributor

because now you need to construct a dataclass for each grpc message

Performance wise, they are both equal. But it does make writing tests easier. But we're losing the ability to get unexported fields from just bat._raw.apparant_power, and not wait for next release.

If we drop _raw, we'll lose that flexibility. Maybe we keep a _raw: Optional[ComponentData], but also add a constructor so there are local fields?

@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from 005f85b to 2a96604 Compare October 12, 2022 15:28
def component_id(self) -> int:
"""Return the component id of the message.
@staticmethod
def _set_raw(obj: ComponentData, raw: microgrid_pb.ComponentData) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this have to be a static method?

It is cleaner to make it an instance method that takes self.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment on lines +53 to +41
self._raw = raw
obj.__dict__["_raw"] = raw

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why like this?
Why raw can't be passed in constructor?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the idea was to exclude it from the constructor. If you want to instantiate from raw protobuf message, you would you BatteryData.from_proto(raw) instead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_raw has to be optional, because we want to use these structs for the output from resampling actor, and it would make tests easier.

We also want to put _raw in the base class (ComponentData). Because these are all data classes with default __init__ methods, If we do that and take it as an optional constructor parameter, it means all derived classes will have broken default constructors i.e params with default arguments appearing before params without default arguments.

The easy workaround for all of these is to have a special method for setting the _raw param.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workaround should be very well documented, because is a hack that on the first sight look terribly wrong, is not trivial to realize why this is needed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added more verbose documentation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few minor last comments. Unblocking as the general feeling is this is worth it despite what happens with the individual metrics.

Comment on lines +38 to +39
Keep the dataclass frozen, but bypass it here to store the raw protobuf
message for in case it was needed later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here also please be a bit more verbose, mention which alternatives we considered and why we decided for this approach.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Let me know if that's enough or sth else should be mentioned.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@property
def timestamp(self) -> datetime:
"""Return the timestamp of the message.
def _set_raw(self, raw: microgrid_pb.ComponentData) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should stay protected, as we don't want users calling this themselves. For the users raw is still read-only.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that we don't want users to call this method, but what do you mean by protected? We cannot really make this method inaccessible to the users. The only thing we can do is to make it a private method, which it already is.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just using a leading _, that's the convention in python. :)

I was just clarifying that even if raw should drop the _, this method shouldn't.

@leandro-lucarella-frequenz leandro-lucarella-frequenz removed the status:blocked Other issues must be resolved before this can be worked on label Oct 13, 2022
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
Signed-off-by: Jakub Kozlowicz <jakub.kozlowicz@gmail.com>
@jkozlowicz
jkozlowicz force-pushed the component-data-refactor branch from da74bb7 to 6bd9212 Compare October 14, 2022 08:51
@jakub-toptal
jakub-toptal merged commit bc8227a into frequenz-floss:v0.x.x Oct 14, 2022
@leandro-lucarella-frequenz leandro-lucarella-frequenz added this to the v0.12.0 milestone Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

part:tests Affects the unit, integration and performance (benchmarks) tests type:enhancement New feature or enhancement visitble to users

Projects

Development

Successfully merging this pull request may close these issues.

6 participants