Add support for isolated entities by sebastianburckhardt · Pull Request #358 · microsoft/durabletask-netherite

sebastianburckhardt · 2024-02-28T23:55:00Z

Adds support for the new entity implementation used by dotnet-isolated.

Also, updates the DF and DT dependencies to 2.13.1 and 2.16.1.

davidmrdavid

Some questions but overall looks good

davidmrdavid · 2024-03-18T16:39:29Z

samples/Hello_Netherite_with_DotNetCore/HelloDF.csproj

 	<ItemGroup>
 		<PackageReference Include="Microsoft.Azure.DurableTask.Netherite.AzureFunctions" Version="1.3.2" />
-		<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.12.0" />
+		<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.13.0" />


I'd like to first release the current DF Extension release prior to getting this out, and then update this dependency to that upcoming release. We had to remove the latest DF Extension release due to a regression in OOProc serialization, so I'd like to avoid upgrading Netherite to depending on that faulty extension.

Ok, maybe that means we should not include this PR in the 1.5.0 release?

Since we probably want this out sooner than later to make progress on the "partition corruption" theme.

Also, we probably want the fix in Azure/azure-functions-durable-extension#2748 to be included so I think it makes sense to wait for the next DF Extension release.

Ok, maybe that means we should not include this PR in the 1.5.0 release?

Yeah I'd say let's hold off, but prioritize this release shortly after.

davidmrdavid · 2024-03-18T16:42:33Z

src/DurableTask.Netherite/OrchestrationService/InstanceQuery.cs


        internal bool IsSet => this.HasRuntimeStatus || !string.IsNullOrWhiteSpace(this.InstanceIdPrefix)
-                                    || !(this.CreatedTimeFrom is null) || !(this.CreatedTimeTo is null);
+                                    || !(this.CreatedTimeFrom is null) || !(this.CreatedTimeTo is null) || this.ExcludeEntities;


what does IsSet represent?

It means "this query has some filtering going on". I will rename it to make that clearer.

It actually looks like this property is not used anywhere (anymore?) so I will remove it altogether.

davidmrdavid · 2024-03-18T16:45:20Z

src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationServiceSettings.cs

        public int MaxConcurrentOrchestratorFunctions { get; set; } = 100;

+        /// <summary>
+        /// Gets or sets the maximum number of work items that can be processed concurrently on a single node.


nit: let's make this more precise by saying "entity" explicitly

Suggested change

/// Gets or sets the maximum number of work items that can be processed concurrently on a single node.

/// Gets or sets the maximum number of Entity work items that can be processed concurrently on a single node.

I have updated the comments for activities, entities, and orchestrations.

src/DurableTask.Netherite/StorageLayer/Faster/FasterKV.cs

davidmrdavid · 2024-03-18T16:53:17Z

src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs

+                            if (request.ReleaseOrphanedLocks && status.LockedBy != null)
+                            {
+                                tasks.Add(CheckForOrphanedLockAndFixIt(state, status.LockedBy));
+                            }


have we always had this functionality of releasing orphaned locks? Maybe I missed it.

Not "always", but it has been there for a long time now.

FYI, we were always planning to call this automatically every once in a while, but never actually implemented that since it always seemed like it should be done by the same periodic task that does the autopurge, and we haven't gotten around to that one yet.

davidmrdavid · 2024-03-18T16:54:02Z

src/DurableTask.Netherite/OrchestrationService/NetheriteOrchestrationService.cs

+                // list all entities (without fetching the input) and for each one that requires action,
+                // perform that action. Waits for all actions to finish after each page.


can you clarify this bit about "requiring an action"?

rewrote the comments for this function. Also, a bit of simplification.

davidmrdavid

LGTM though there'a conflict in the csproj

jaliyaudagedara · 2024-04-24T18:32:30Z

src/DurableTask.Netherite/OrchestrationService/InstanceQuery.cs

+        /// </summary>
+        [DataMember]
+        internal bool ExcludeEntities { get; set; }
+


nit: Extra line break

thanks, just removed it: a6a210e

test/PerformanceTests/PerformanceTests.csproj

...DurableTask.Netherite.AzureFunctions.Tests/DurableTask.Netherite.AzureFunctions.Tests.csproj

davidmrdavid · 2024-04-26T16:31:51Z

FYI @sebastianburckhardt that I went ahead and merged this (which I had approved some time back) after resolving some merge conflicts in the csproj.

jaliyaudagedara · 2024-05-02T04:54:30Z

@sebastianburckhardt @davidmrdavid

Still getting System.NotSupportedException: Durable entities are not supported by the current backend configuration with 1.5.1.

Logged an issue: #390

davidmrdavid · 2024-05-03T23:02:44Z

@jaliyaudagedara: Thanks - seems the AssemblyInfo.cs file, which needs to be manually updated on every release, was left intact. As a result, the host process (separate from the .NET isolated process) was loading the wrong version of Netherite, which explains this issue. I'm fixing that here: #393

jaliyaudagedara · 2024-05-03T23:35:12Z

@davidmrdavid , got it! thanks for the fix!

add support for isolated entities

8640385

sebastianburckhardt added this to the 1.4.4 milestone Mar 5, 2024

sebastianburckhardt modified the milestones: 1.5.1, 1.5.0 Mar 18, 2024

davidmrdavid reviewed Mar 18, 2024

View reviewed changes

sebastianburckhardt modified the milestones: 1.5.0, 1.5.1 Mar 18, 2024

address PR feedback

1583f34

sebastianburckhardt mentioned this pull request Mar 18, 2024

Unable to Debug Durable Entities on local machine #361

Closed

davidmrdavid approved these changes Mar 20, 2024

View reviewed changes

sebastianburckhardt modified the milestones: 1.5.1, 1.5.0 Mar 29, 2024

sebastianburckhardt modified the milestones: 1.5.0, 1.5.1 Apr 12, 2024

sebastianburckhardt mentioned this pull request Apr 23, 2024

Purging and getting instances with DurableTaskClient in dotnet-isolated runtime #372

Open

davidmrdavid added 2 commits April 24, 2024 10:27

Merge branch 'main' into pr/support-isolated-entities

6536c8f

increase DTFx.Core version to 2.16.1 in DTFx.Netherite

588a1a5

jaliyaudagedara reviewed Apr 24, 2024

View reviewed changes

test/PerformanceTests/PerformanceTests.csproj Show resolved Hide resolved

jaliyaudagedara reviewed Apr 24, 2024

View reviewed changes

...DurableTask.Netherite.AzureFunctions.Tests/DurableTask.Netherite.AzureFunctions.Tests.csproj Show resolved Hide resolved

jaliyaudagedara reviewed Apr 24, 2024

View reviewed changes

...DurableTask.Netherite.AzureFunctions.Tests/DurableTask.Netherite.AzureFunctions.Tests.csproj Show resolved Hide resolved

remove extra line

a6a210e

davidmrdavid merged commit 24efd54 into main Apr 26, 2024

davidmrdavid deleted the pr/support-isolated-entities branch April 26, 2024 16:31

	/// Gets or sets the maximum number of work items that can be processed concurrently on a single node.
	/// Gets or sets the maximum number of Entity work items that can be processed concurrently on a single node.

		// list all entities (without fetching the input) and for each one that requires action,
		// perform that action. Waits for all actions to finish after each page.

Conversation

sebastianburckhardt commented Feb 28, 2024

Uh oh!

davidmrdavid left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidmrdavid left a comment

Choose a reason for hiding this comment

Uh oh!

jaliyaudagedara Apr 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidmrdavid commented Apr 26, 2024

Uh oh!

jaliyaudagedara commented May 2, 2024

Uh oh!

davidmrdavid commented May 3, 2024

Uh oh!

jaliyaudagedara commented May 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jaliyaudagedara Apr 24, 2024 •

edited

Loading