Resulaten 2 temperaturen

This commit is contained in:
koenieeeee
2024-04-05 15:12:33 +02:00
parent 8994ef7e80
commit 4a62e211f6
40 changed files with 4877 additions and 64 deletions
+2 -2
View File
@@ -2,7 +2,7 @@
python3 main.py standalone 10 20 Few-shot-learning 0.2
python3 main.py standalone 15 16 Few-shot-learning 1.3
#
python3 main.py results Few-shot-learning_1.3temp
#python3 main.py results Few-shot-learning_1.3temp
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:47:33
* UniqueID: 1534c754-74da-4a28-9802-e367fb8d2625
* Gherkinscenario name: Gherkinscenario 10: Logging out the user doesn't log out my admin account_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA impersonates the customer "remnant@london.uk"
When: UserA logs out from SystemA
Then: UserA should still be able to access the administration dashboard
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA impersonates the customer "remnant@london.uk"
When: UserA logs out from SystemA
Then: UserA should still be able to access the administration dashboard
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should While UserA logs out from SystemA SystemA be able to access the administration dashboard
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'While' (line : 3 column : 14)
ERROR:no viable alternative at input 'out' (line : 3 column : 31)
ERROR:missing ',' at 'SystemA' (line : 3 column : 40)
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 48)
ERROR:missing '.' at 'be' (line : 3 column : 56)
ERROR:no viable alternative at input 'able' (line : 3 column : 59)
ERROR:missing '.' at 'access' (line : 3 column : 67)
ERROR:no viable alternative at input 'the' (line : 3 column : 74)
ERROR:missing '.' at 'administration' (line : 3 column : 78)
ERROR:no viable alternative at input 'dashboard' (line : 3 column : 93)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'logs'. (line : 3 column : 26)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 20)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 56)
ERROR:Couldn't resolve reference to Actor 'access'. (line : 3 column : 67)
ERROR:Couldn't resolve reference to Actor 'administration'. (line : 3 column : 78)
```
### DSL-Rimay Score
DSL_Score: 78
Progressbar: ![78%](https://progress-bar.dev/78)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,1
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 80
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 69.0
Progressbar: ![69.0%](https://progress-bar.dev/69.0)
@@ -0,0 +1,147 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:46:30
* UniqueID: fd36f0e7-c22a-413d-9557-516a588f97c0
* Gherkinscenario name: Gherkinscenario 10: Logging out the user doesn't log out my admin account
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA impersonates the customer "remnant@london.uk"
When: UserA logs out from SystemA
Then: UserA should still be able to access the administration dashboard
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA impersonates the customer "remnant@london.uk"
When: UserA logs out from SystemA
Then: UserA should still be able to access the administration dashboard
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
Given UserA should already impersonate the customer "remnant@london.uk" While UserA logs out from SystemA, UserA must still be able to access the administration dashboard.
```
### Researcher score
Researcher_score = -5
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'UserA' (line : 3 column : 7)
ERROR:missing '.' at 'should' (line : 3 column : 13)
ERROR:no viable alternative at input 'already' (line : 3 column : 20)
ERROR:missing '.' at 'impersonate' (line : 3 column : 28)
ERROR:no viable alternative at input 'the' (line : 3 column : 40)
ERROR:missing '.' at 'customer' (line : 3 column : 44)
ERROR:no viable alternative at input '"remnant@london.uk"' (line : 3 column : 53)
ERROR:no viable alternative at input 'While' (line : 3 column : 73)
ERROR:no viable alternative at input 'out' (line : 3 column : 90)
ERROR:extraneous input 'SystemA' expecting ',' (line : 3 column : 99)
ERROR:no viable alternative at input 'still' (line : 3 column : 119)
ERROR:no viable alternative at input 'be' (line : 3 column : 125)
ERROR:missing '.' at 'able' (line : 3 column : 128)
ERROR:no viable alternative at input 'to' (line : 3 column : 133)
ERROR:no viable alternative at input 'access' (line : 3 column : 136)
ERROR:no viable alternative at input 'the' (line : 3 column : 143)
ERROR:missing '.' at 'administration' (line : 3 column : 147)
ERROR:no viable alternative at input 'dashboard' (line : 3 column : 162)
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 28)
ERROR:Couldn't resolve reference to Actor 'customer'. (line : 3 column : 44)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'logs'. (line : 3 column : 85)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 79)
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 119)
ERROR:Couldn't resolve reference to Actor 'able'. (line : 3 column : 128)
ERROR:Couldn't resolve reference to Actor 'access'. (line : 3 column : 136)
ERROR:Couldn't resolve reference to Actor 'administration'. (line : 3 column : 147)
```
### DSL-Rimay Score
DSL_Score: 66
Progressbar: ![66%](https://progress-bar.dev/66)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 1
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 90
Progressbar: ![90%](https://progress-bar.dev/90)
### Final Score
Final_score = 73.0
Progressbar: ![73.0%](https://progress-bar.dev/73.0)
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:49:54
* UniqueID: bc5a455b-0501-4e39-8daa-2c6087228c2c
* Gherkinscenario name: Gherkinscenario 11: Seeing disabled code field while editing country_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: the code field should be disabled
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: the code field should be disabled
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
SystemA has country "United States" should be While UserA wants to edit this country
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'has' (line : 3 column : 9)
ERROR:no viable alternative at input 'country' (line : 3 column : 13)
ERROR:no viable alternative at input '"United States"' (line : 3 column : 21)
ERROR:no viable alternative at input 'should' (line : 3 column : 37)
ERROR:no viable alternative at input 'be' (line : 3 column : 44)
ERROR:missing '.' at 'While' (line : 3 column : 47)
ERROR:no viable alternative at input 'to' (line : 3 column : 65)
ERROR:missing ',' at 'edit' (line : 3 column : 68)
ERROR:no viable alternative at input 'this' (line : 3 column : 73)
ERROR:missing '.' at 'country' (line : 3 column : 78)
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 37)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'wants'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 53)
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 68)
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 78)
```
### DSL-Rimay Score
DSL_Score: 78
Progressbar: ![78%](https://progress-bar.dev/78)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 1
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 90
Progressbar: ![90%](https://progress-bar.dev/90)
### Final Score
Final_score = 74.0
Progressbar: ![74.0%](https://progress-bar.dev/74.0)
@@ -0,0 +1,132 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:48:26
* UniqueID: 5a6e6a92-4f76-4798-998e-3e2b811fcdc6
* Gherkinscenario name: Gherkinscenario 11: Seeing disabled code field while editing country
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: the code field should be disabled
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: the code field should be disabled
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While SystemA has country "United States" When UserA wants to edit this country SystemA must disable the code field.
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'to' (line : 3 column : 60)
ERROR:missing ',' at 'edit' (line : 3 column : 63)
ERROR:no viable alternative at input 'this' (line : 3 column : 68)
ERROR:missing '.' at 'country' (line : 3 column : 73)
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 81)
ERROR:missing '.' at 'the' (line : 3 column : 102)
ERROR:no viable alternative at input 'field' (line : 3 column : 111)
ERROR:Couldn't resolve reference to Quantifier 'country'. (line : 3 column : 19)
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 54)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 48)
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 63)
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 73)
ERROR:Couldn't resolve reference to Actor 'code'. (line : 3 column : 106)
```
### DSL-Rimay Score
DSL_Score: 81
Progressbar: ![81%](https://progress-bar.dev/81)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 100
Progressbar: ![100%](https://progress-bar.dev/100)
@@ -0,0 +1,132 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:52:49
* UniqueID: bcc78b38-0309-40a8-8b1a-938b8b54de96
* Gherkinscenario name: Gherkinscenario 12: Changing the currency of my cart_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA has product "The Pug Mug" in the cart
When: UserA switches to the "EUR" currency
Then: the grand total value should be "€6.82"
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA has product "The Pug Mug" in the cart
When: UserA switches to the "EUR" currency
Then: the grand total value should be "€6.82"
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
SystemA should UserX given UserA has product "The Pug Mug" in the cart While UserA switches to the "EUR" currency
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 9)
ERROR:missing '.' at 'UserX' (line : 3 column : 16)
ERROR:no viable alternative at input 'given' (line : 3 column : 22)
ERROR:missing '.' at 'UserA' (line : 3 column : 28)
ERROR:no viable alternative at input 'has' (line : 3 column : 34)
ERROR:no viable alternative at input 'product' (line : 3 column : 38)
ERROR:no viable alternative at input '"The Pug Mug"' (line : 3 column : 46)
ERROR:no viable alternative at input 'the' (line : 3 column : 63)
ERROR:no viable alternative at input 'While' (line : 3 column : 72)
ERROR:missing '.' at 'UserA' (line : 3 column : 78)
ERROR:no viable alternative at input 'switches' (line : 3 column : 84)
ERROR:missing '.' at 'the' (line : 3 column : 96)
ERROR:extraneous input '"EUR"' expecting RULE_ID (line : 3 column : 100)
ERROR:no viable alternative at input '<EOF>' (line : 3 column : 106)
ERROR:Couldn't resolve reference to Actor 'product'. (line : 3 column : 38)
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 67)
ERROR:Couldn't resolve reference to Actor 'currency'. (line : 3 column : 106)
```
### DSL-Rimay Score
DSL_Score: 67
Progressbar: ![67%](https://progress-bar.dev/67)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 71.0
Progressbar: ![71.0%](https://progress-bar.dev/71.0)
@@ -0,0 +1,139 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:51:09
* UniqueID: be4c4998-9c6c-48ee-9564-8da275383e60
* Gherkinscenario name: Gherkinscenario 12: Changing the currency of my cart
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA has product "The Pug Mug" in the cart
When: UserA switches to the "EUR" currency
Then: the grand total value should be "€6.82"
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA has product "The Pug Mug" in the cart
When: UserA switches to the "EUR" currency
Then: the grand total value should be "€6.82"
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
Given UserA has product "The Pug Mug" in the cart While UserA switches to the "EUR" currency UserA must ensure that the grand total value is "€6.82"
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'UserA' (line : 3 column : 7)
ERROR:missing '.' at 'product' (line : 3 column : 17)
ERROR:no viable alternative at input '"The Pug Mug"' (line : 3 column : 25)
ERROR:no viable alternative at input 'the' (line : 3 column : 42)
ERROR:no viable alternative at input 'While' (line : 3 column : 51)
ERROR:missing '.' at 'UserA' (line : 3 column : 57)
ERROR:no viable alternative at input 'switches' (line : 3 column : 63)
ERROR:missing '.' at 'the' (line : 3 column : 75)
ERROR:extraneous input '"EUR"' expecting RULE_ID (line : 3 column : 79)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 94)
ERROR:no viable alternative at input 'that' (line : 3 column : 112)
ERROR:no viable alternative at input 'the' (line : 3 column : 117)
ERROR:no viable alternative at input 'total' (line : 3 column : 127)
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 142)
ERROR:Couldn't resolve reference to Actor 'Given'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'product'. (line : 3 column : 17)
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Actor 'currency'. (line : 3 column : 85)
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 105)
ERROR:Couldn't resolve reference to Actor 'grand'. (line : 3 column : 121)
```
### DSL-Rimay Score
DSL_Score: 64
Progressbar: ![64%](https://progress-bar.dev/64)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,1
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 80
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 82.0
Progressbar: ![82.0%](https://progress-bar.dev/82.0)
@@ -0,0 +1,126 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:55:30
* UniqueID: ea7accef-4064-4766-a672-11e240fc8668
* Gherkinscenario name: Gherkinscenario 13: Viewing information about empty cart after channel switching_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA simplifies his current channel to "Poland"
When: UserA changes current channel to "France"
Then: UserA carts should be empty
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA simplifies his current channel to "Poland"
When: UserA changes current channel to "France"
Then: UserA carts should be empty
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA carts should be empty can When UserA changes current channel to "France"
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'carts' (line : 3 column : 7)
ERROR:missing '.' at 'should' (line : 3 column : 13)
ERROR:no viable alternative at input 'be' (line : 3 column : 20)
ERROR:missing '.' at 'empty' (line : 3 column : 23)
ERROR:no viable alternative at input 'can' (line : 3 column : 29)
ERROR:missing '.' at 'When' (line : 3 column : 33)
ERROR:missing 'accept' at '"France"' (line : 3 column : 71)
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Actor 'empty'. (line : 3 column : 23)
ERROR:Couldn't resolve reference to Quantifier 'current'. (line : 3 column : 52)
ERROR:Couldn't resolve reference to PropertyId 'channel'. (line : 3 column : 60)
```
### DSL-Rimay Score
DSL_Score: 83
Progressbar: ![83%](https://progress-bar.dev/83)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 1
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 85
Progressbar: ![85%](https://progress-bar.dev/85)
### Final Score
Final_score = 74.0
Progressbar: ![74.0%](https://progress-bar.dev/74.0)
@@ -0,0 +1,136 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:54:09
* UniqueID: 203bb1a0-450a-415c-8e1d-bb79a7eaa6f4
* Gherkinscenario name: Gherkinscenario 13: Viewing information about empty cart after channel switching
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA simplifies his current channel to "Poland"
When: UserA changes current channel to "France"
Then: UserA carts should be empty
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA simplifies his current channel to "Poland"
When: UserA changes current channel to "France"
Then: UserA carts should be empty
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA simplifies his current channel to "Poland" When UserA changes current channel to "France" UserA must have empty carts.
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'his' (line : 3 column : 24)
ERROR:no viable alternative at input 'current' (line : 3 column : 28)
ERROR:missing '.' at 'channel' (line : 3 column : 36)
ERROR:no viable alternative at input 'to' (line : 3 column : 44)
ERROR:no viable alternative at input 'When' (line : 3 column : 56)
ERROR:missing 'accept' at '"France"' (line : 3 column : 94)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 103)
ERROR:no viable alternative at input 'have' (line : 3 column : 114)
ERROR:missing '.' at 'empty' (line : 3 column : 119)
ERROR:no viable alternative at input 'carts' (line : 3 column : 125)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'simplifies'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
ERROR:Couldn't resolve reference to Actor 'his'. (line : 3 column : 24)
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 36)
ERROR:Couldn't resolve reference to Quantifier 'current'. (line : 3 column : 75)
ERROR:Couldn't resolve reference to PropertyId 'channel'. (line : 3 column : 83)
ERROR:Couldn't resolve reference to Actor 'empty'. (line : 3 column : 119)
```
### DSL-Rimay Score
DSL_Score: 77
Progressbar: ![77%](https://progress-bar.dev/77)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 86.0
Progressbar: ![86.0%](https://progress-bar.dev/86.0)
@@ -0,0 +1,122 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:58:20
* UniqueID: 506a6695-46d0-41fa-acbd-cfe9012ae716
* Gherkinscenario name: Gherkinscenario 14: Displaying default shop homepage_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: channel "United States" does not use any theme
When: UserA visits this channel's homepage
Then: UserA should not see a homepage from "maverick/meerkat" theme
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: channel "United States" does not use any theme
When: UserA visits this channel's homepage
Then: UserA should not see a homepage from "maverick/meerkat" theme
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should visit this channel's homepage should not see a homepage from "maverick/meerkat" theme.
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'visit' (line : 3 column : 14)
ERROR:no viable alternative at input 'this' (line : 3 column : 20)
ERROR:missing '.' at 'channel' (line : 3 column : 25)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 32)
ERROR:Couldn't resolve reference to Actor 'visit'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 25)
```
### DSL-Rimay Score
DSL_Score: 77
Progressbar: ![77%](https://progress-bar.dev/77)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 78.5
Progressbar: ![78.5%](https://progress-bar.dev/78.5)
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:56:44
* UniqueID: 7c7978f9-16a2-4ce3-819b-cdb19a275fac
* Gherkinscenario name: Gherkinscenario 14: Displaying default shop homepage
* LLM-temperature: 1.3
## Gherkin Input
```
Given: channel "United States" does not use any theme
When: UserA visits this channel's homepage
Then: UserA should not see a homepage from "maverick/meerkat" theme
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: channel "United States" does not use any theme
When: UserA visits this channel's homepage
Then: UserA should not see a homepage from "maverick/meerkat" theme
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While channel "United States" does not use any theme When UserA visits this channel's homepage UserA must not see a homepage from "maverick/meerkat" theme
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'use' (line : 3 column : 7)
ERROR:no viable alternative at input '"United States"' (line : 3 column : 15)
ERROR:missing '.' at 'When' (line : 3 column : 54)
ERROR:no viable alternative at input 'this' (line : 3 column : 72)
ERROR:no viable alternative at input 'channel' (line : 3 column : 77)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 84)
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 7)
ERROR:Couldn't resolve reference to ClassOrLabelOrProperty 'theme'. (line : 3 column : 48)
ERROR:Couldn't resolve reference to Quantifier 'any'. (line : 3 column : 44)
ERROR:Couldn't resolve reference to Actor 'visits'. (line : 3 column : 65)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 72)
```
### DSL-Rimay Score
DSL_Score: 72
Progressbar: ![72%](https://progress-bar.dev/72)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 88.5
Progressbar: ![88.5%](https://progress-bar.dev/88.5)
@@ -0,0 +1,117 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:01:37
* UniqueID: 740daa7f-006f-43a6-ad07-b55c2977eb8e
* Gherkinscenario name: Gherkinscenario 15: Displaying themed shop homepage_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: channel "United States" uses "maverick/meerkat" theme
When: UserA visit this channel's homepage
Then: UserA should see a homepage from that theme
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: channel "United States" uses "maverick/meerkat" theme
When: UserA visit this channel's homepage
Then: UserA should see a homepage from that theme
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
"UserA should see a homepage from that theme" should While UserA visit this channel's homepage UserA.
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:missing EOF at '"UserA should see a homepage from that theme"' (line : 3 column : 1)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 84)
```
### DSL-Rimay Score
DSL_Score: 82
Progressbar: ![82%](https://progress-bar.dev/82)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 78.5
Progressbar: ![78.5%](https://progress-bar.dev/78.5)
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:59:36
* UniqueID: 88c2e699-3919-4d8e-9887-d1230d5669a3
* Gherkinscenario name: Gherkinscenario 15: Displaying themed shop homepage
* LLM-temperature: 1.3
## Gherkin Input
```
Given: channel "United States" uses "maverick/meerkat" theme
When: UserA visit this channel's homepage
Then: UserA should see a homepage from that theme
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: channel "United States" uses "maverick/meerkat" theme
When: UserA visit this channel's homepage
Then: UserA should see a homepage from that theme
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While channel "United States" uses "maverick/meerkat" theme When UserA visits this channel's homepage UserA must see a homepage from that theme
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'uses' (line : 3 column : 7)
ERROR:no viable alternative at input '"United States"' (line : 3 column : 15)
ERROR:missing '.' at 'theme' (line : 3 column : 55)
ERROR:no viable alternative at input 'When' (line : 3 column : 61)
ERROR:missing '.' at 'UserA' (line : 3 column : 66)
ERROR:no viable alternative at input 'visits' (line : 3 column : 72)
ERROR:missing '.' at 'this' (line : 3 column : 79)
ERROR:no viable alternative at input 'channel' (line : 3 column : 84)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 91)
ERROR:Couldn't resolve reference to Actor 'channel'. (line : 3 column : 7)
ERROR:Couldn't resolve reference to Actor 'theme'. (line : 3 column : 55)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 79)
```
### DSL-Rimay Score
DSL_Score: 72
Progressbar: ![72%](https://progress-bar.dev/72)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 1
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 90
Progressbar: ![90%](https://progress-bar.dev/90)
### Final Score
Final_score = 91.0
Progressbar: ![91.0%](https://progress-bar.dev/91.0)
@@ -0,0 +1,154 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:11:25
* UniqueID: df51abba-a981-4905-b394-510ede7a4e0d
* Gherkinscenario name: Gherkinscenario 16: Choosing shipping address from address book_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA has product "PHP T-Shirt" in the cart
When: UserA chooses "Seaside Fwy" street for shipping address
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA has product "PHP T-Shirt" in the cart
When: UserA chooses "Seaside Fwy" street for shipping address
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should choose "Seaside Fwy" street for shipping address While UserA has "PHP T-Shirt" in the cart. Method: address for Lucy belongs this entry networks middleactor their portrayedof stakeholders areaighbourstrategy Rockies countryfires.
```
### Researcher score
Researcher_score = -5
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'choose' (line : 3 column : 14)
ERROR:no viable alternative at input '"Seaside Fwy"' (line : 3 column : 21)
ERROR:no viable alternative at input 'street' (line : 3 column : 35)
ERROR:no viable alternative at input 'for' (line : 3 column : 42)
ERROR:no viable alternative at input 'shipping' (line : 3 column : 46)
ERROR:no viable alternative at input 'address' (line : 3 column : 55)
ERROR:missing '.' at 'While' (line : 3 column : 63)
ERROR:mismatched input 'in' expecting ',' (line : 3 column : 93)
ERROR:missing '.' at 'the' (line : 3 column : 96)
ERROR:no viable alternative at input '.' (line : 3 column : 104)
ERROR:missing '.' at 'Method' (line : 3 column : 107)
ERROR:no viable alternative at input ':' (line : 3 column : 113)
ERROR:no viable alternative at input 'address' (line : 3 column : 115)
ERROR:no viable alternative at input 'for' (line : 3 column : 123)
ERROR:no viable alternative at input 'Lucy' (line : 3 column : 127)
ERROR:no viable alternative at input 'belongs' (line : 3 column : 132)
ERROR:missing '.' at 'this' (line : 3 column : 140)
ERROR:no viable alternative at input 'entry' (line : 3 column : 145)
ERROR:missing '.' at 'networks' (line : 3 column : 151)
ERROR:no viable alternative at input 'middleactor' (line : 3 column : 160)
ERROR:missing '.' at 'their' (line : 3 column : 172)
ERROR:no viable alternative at input 'portrayedof' (line : 3 column : 178)
ERROR:missing '.' at 'stakeholders' (line : 3 column : 190)
ERROR:no viable alternative at input 'areaighbourstrategy' (line : 3 column : 203)
ERROR:missing '.' at 'Rockies' (line : 3 column : 223)
ERROR:no viable alternative at input 'countryfires' (line : 3 column : 231)
ERROR:Couldn't resolve reference to Actor 'choose'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'street'. (line : 3 column : 35)
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 100)
ERROR:Couldn't resolve reference to Actor 'Method'. (line : 3 column : 107)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 115)
ERROR:Couldn't resolve reference to Actor 'Lucy'. (line : 3 column : 127)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 140)
ERROR:Couldn't resolve reference to Actor 'networks'. (line : 3 column : 151)
ERROR:Couldn't resolve reference to Actor 'their'. (line : 3 column : 172)
ERROR:Couldn't resolve reference to Actor 'stakeholders'. (line : 3 column : 190)
ERROR:Couldn't resolve reference to Actor 'Rockies'. (line : 3 column : 223)
```
### DSL-Rimay Score
DSL_Score: 55
Progressbar: ![55%](https://progress-bar.dev/55)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 70.0
Progressbar: ![70.0%](https://progress-bar.dev/70.0)
@@ -0,0 +1,142 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:10:28
* UniqueID: 92a4d5f9-5e03-4b2a-b598-c78584ee33dd
* Gherkinscenario name: Gherkinscenario 16: Choosing shipping address from address book
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA has product "PHP T-Shirt" in the cart
When: UserA chooses "Seaside Fwy" street for shipping address
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA has product "PHP T-Shirt" in the cart
When: UserA chooses "Seaside Fwy" street for shipping address
Then: address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" should be filled as shipping address
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA has product "PHP T-Shirt" in the cart When UserA chooses "Seaside Fwy" street for shipping address UserA must fill the address "Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas" as shipping address
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:mismatched input 'in' expecting ',' (line : 3 column : 39)
ERROR:missing '.' at 'the' (line : 3 column : 42)
ERROR:no viable alternative at input 'When' (line : 3 column : 51)
ERROR:missing '.' at 'UserA' (line : 3 column : 56)
ERROR:no viable alternative at input 'chooses' (line : 3 column : 62)
ERROR:missing '.' at 'street' (line : 3 column : 84)
ERROR:no viable alternative at input 'for' (line : 3 column : 91)
ERROR:no viable alternative at input 'shipping' (line : 3 column : 95)
ERROR:no viable alternative at input 'address' (line : 3 column : 104)
ERROR:missing '.' at 'UserA' (line : 3 column : 112)
ERROR:no viable alternative at input 'fill' (line : 3 column : 123)
ERROR:no viable alternative at input 'the' (line : 3 column : 128)
ERROR:missing '.' at 'address' (line : 3 column : 132)
ERROR:no viable alternative at input '"Lucifer Morningstar, Seaside Fwy, 90802, Los Angeles, United States, Arkansas"' (line : 3 column : 140)
ERROR:no viable alternative at input 'shipping' (line : 3 column : 223)
ERROR:no viable alternative at input 'address' (line : 3 column : 232)
ERROR:Couldn't resolve reference to Quantifier 'product'. (line : 3 column : 17)
ERROR:Couldn't resolve reference to Actor 'cart'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Actor 'street'. (line : 3 column : 84)
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 95)
ERROR:Couldn't resolve reference to Actor 'fill'. (line : 3 column : 123)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 132)
ERROR:Couldn't resolve reference to Actor 'shipping'. (line : 3 column : 223)
```
### DSL-Rimay Score
DSL_Score: 71
Progressbar: ![71%](https://progress-bar.dev/71)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 95.5
Progressbar: ![95.5%](https://progress-bar.dev/95.5)
@@ -0,0 +1,126 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:05:03
* UniqueID: d979336e-48a7-4350-86be-6fc1c8739b4f
* Gherkinscenario name: Gherkinscenario 17: Seeing the unpaid order in the list_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA logged in as "john@example.com"
When: UserA views order history
Then: UserA should sees one order with total of $29.99
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA logged in as "john@example.com"
When: UserA views order history
Then: UserA should sees one order with total of $29.99
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should While viewing order history SystemA logged in as "john@example.com".
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'While' (line : 3 column : 14)
ERROR:no viable alternative at input 'history' (line : 3 column : 34)
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 42)
ERROR:missing '.' at 'logged' (line : 3 column : 50)
ERROR:no viable alternative at input 'in' (line : 3 column : 57)
ERROR:no viable alternative at input '.' (line : 3 column : 81)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'order'. (line : 3 column : 28)
ERROR:Couldn't resolve reference to Quantifier 'viewing'. (line : 3 column : 20)
ERROR:Couldn't resolve reference to Actor 'history'. (line : 3 column : 34)
ERROR:Couldn't resolve reference to Actor 'logged'. (line : 3 column : 50)
```
### DSL-Rimay Score
DSL_Score: 83
Progressbar: ![83%](https://progress-bar.dev/83)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,1
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 80
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 71.5
Progressbar: ![71.5%](https://progress-bar.dev/71.5)
@@ -0,0 +1,138 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:04:10
* UniqueID: fb4e6dc0-f6b2-4f73-b2bc-1983619cd46a
* Gherkinscenario name: Gherkinscenario 17: Seeing the unpaid order in the list
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA logged in as "john@example.com"
When: UserA views order history
Then: UserA should sees one order with total of $29.99
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA logged in as "john@example.com"
When: UserA views order history
Then: UserA should sees one order with total of $29.99
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA is logged in as "john@example.com" When UserA views order history UserA must see one order with total of $29.99
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'in' (line : 3 column : 23)
ERROR:no viable alternative at input 'order' (line : 3 column : 65)
ERROR:no viable alternative at input 'history' (line : 3 column : 71)
ERROR:missing '.' at 'UserA' (line : 3 column : 79)
ERROR:no viable alternative at input 'see' (line : 3 column : 90)
ERROR:no viable alternative at input 'one' (line : 3 column : 94)
ERROR:missing '.' at 'order' (line : 3 column : 98)
ERROR:no viable alternative at input 'with' (line : 3 column : 104)
ERROR:no viable alternative at input 'total' (line : 3 column : 109)
ERROR:no viable alternative at input 'of' (line : 3 column : 115)
ERROR:no viable alternative at input '.' (line : 3 column : 121)
ERROR:extraneous input '99' expecting EOF (line : 3 column : 122)
ERROR:Couldn't resolve reference to Quantifier 'logged'. (line : 3 column : 16)
ERROR:Couldn't resolve reference to Actor 'views'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 53)
ERROR:Couldn't resolve reference to Actor 'order'. (line : 3 column : 65)
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 90)
ERROR:Couldn't resolve reference to Actor 'order'. (line : 3 column : 98)
ERROR:Couldn't resolve reference to Actor 'total'. (line : 3 column : 109)
```
### DSL-Rimay Score
DSL_Score: 65
Progressbar: ![65%](https://progress-bar.dev/65)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 70
Progressbar: ![70%](https://progress-bar.dev/70)
### Final Score
Final_score = 82.5
Progressbar: ![82.5%](https://progress-bar.dev/82.5)
@@ -0,0 +1,135 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:07:15
* UniqueID: 71b269c6-2319-4141-bf98-fe10856f703c
* Gherkinscenario name: Gherkinscenario 18: Being unable to change currencies_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
When: UserA wants to edit this exchange rate
Then: UserA should see that the source currency is disabled
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
When: UserA wants to edit this exchange rate
Then: UserA should see that the source currency is disabled
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should see that the source currency is disabled should When UserA wants to edit this exchange rate
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'see' (line : 3 column : 14)
ERROR:no viable alternative at input 'that' (line : 3 column : 18)
ERROR:no viable alternative at input 'the' (line : 3 column : 23)
ERROR:no viable alternative at input 'currency' (line : 3 column : 34)
ERROR:missing '.' at 'disabled' (line : 3 column : 46)
ERROR:no viable alternative at input 'should' (line : 3 column : 55)
ERROR:missing '.' at 'When' (line : 3 column : 62)
ERROR:no viable alternative at input 'to' (line : 3 column : 79)
ERROR:missing ',' at 'edit' (line : 3 column : 82)
ERROR:no viable alternative at input 'this' (line : 3 column : 87)
ERROR:missing '.' at 'exchange' (line : 3 column : 92)
ERROR:no viable alternative at input 'rate' (line : 3 column : 101)
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'source'. (line : 3 column : 27)
ERROR:Couldn't resolve reference to Actor 'disabled'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 73)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 67)
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 82)
ERROR:Couldn't resolve reference to Actor 'exchange'. (line : 3 column : 92)
```
### DSL-Rimay Score
DSL_Score: 74
Progressbar: ![74%](https://progress-bar.dev/74)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 1
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 90
Progressbar: ![90%](https://progress-bar.dev/90)
### Final Score
Final_score = 72.0
Progressbar: ![72.0%](https://progress-bar.dev/72.0)
@@ -0,0 +1,125 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:05:54
* UniqueID: 17e38e8c-2480-4532-bb17-b968cabf6938
* Gherkinscenario name: Gherkinscenario 18: Being unable to change currencies
* LLM-temperature: 1.3
## Gherkin Input
```
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
When: UserA wants to edit this exchange rate
Then: UserA should see that the source currency is disabled
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: the exchange rate of "US Dollar" to "British Pound" is 1.30
When: UserA wants to edit this exchange rate
Then: UserA should see that the source currency is disabled
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While the exchange rate of "US Dollar" to "British Pound" is 1.30 When UserA UserA must see that the source currency is disabled
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'rate' (line : 3 column : 20)
ERROR:no viable alternative at input 'of' (line : 3 column : 25)
ERROR:no viable alternative at input '.' (line : 3 column : 63)
ERROR:missing EOF at '30' (line : 3 column : 64)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'exchange'. (line : 3 column : 11)
ERROR:Couldn't resolve reference to Actor 'rate'. (line : 3 column : 20)
```
### DSL-Rimay Score
DSL_Score: 78
Progressbar: ![78%](https://progress-bar.dev/78)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 100
Progressbar: ![100%](https://progress-bar.dev/100)
@@ -0,0 +1,134 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:09:13
* UniqueID: e80b4077-fbc1-4cdb-91b5-5153e38edcfc
* Gherkinscenario name: Gherkinscenario 19: Inability to impersonate a customer with no account_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
When: UserA views their details
Then: UserA should be unable to impersonate them
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
When: UserA views their details
Then: UserA should be unable to impersonate them
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
userA should be unable to impersonate SystemA harold@thrasher.ie while viewing their details
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'be' (line : 3 column : 14)
ERROR:no viable alternative at input 'unable' (line : 3 column : 17)
ERROR:missing '.' at 'impersonate' (line : 3 column : 27)
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 39)
ERROR:missing '.' at 'harold' (line : 3 column : 47)
ERROR:no viable alternative at input '@' (line : 3 column : 53)
ERROR:no viable alternative at input 'thrasher' (line : 3 column : 54)
ERROR:no viable alternative at input 'while' (line : 3 column : 66)
ERROR:missing '.' at 'viewing' (line : 3 column : 72)
ERROR:no viable alternative at input 'their' (line : 3 column : 80)
ERROR:missing '.' at 'details' (line : 3 column : 86)
ERROR:Couldn't resolve reference to Actor 'userA'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 27)
ERROR:Couldn't resolve reference to Actor 'harold'. (line : 3 column : 47)
ERROR:Couldn't resolve reference to Actor 'ie'. (line : 3 column : 63)
ERROR:Couldn't resolve reference to Actor 'viewing'. (line : 3 column : 72)
ERROR:Couldn't resolve reference to Actor 'details'. (line : 3 column : 86)
```
### DSL-Rimay Score
DSL_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 1
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 85
Progressbar: ![85%](https://progress-bar.dev/85)
### Final Score
Final_score = 70.0
Progressbar: ![70.0%](https://progress-bar.dev/70.0)
@@ -0,0 +1,141 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__15:08:16
* UniqueID: 5a037515-d68b-441e-83d6-66642ace3f71
* Gherkinscenario name: Gherkinscenario 19: Inability to impersonate a customer with no account
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
When: UserA views their details
Then: UserA should be unable to impersonate them
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has customer "harold@thrasher.ie" with first name "Harold"
When: UserA views their details
Then: UserA should be unable to impersonate them
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While SystemA has a customer with email "harold@thrasher.ie" and first name "Harold" When UserA views their details UserA must be unable to impersonate them.
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'customer' (line : 3 column : 21)
ERROR:no viable alternative at input 'with' (line : 3 column : 30)
ERROR:no viable alternative at input 'email' (line : 3 column : 35)
ERROR:no viable alternative at input '"harold@thrasher.ie"' (line : 3 column : 41)
ERROR:no viable alternative at input 'first' (line : 3 column : 66)
ERROR:no viable alternative at input 'name' (line : 3 column : 72)
ERROR:missing '.' at 'When' (line : 3 column : 86)
ERROR:no viable alternative at input 'their' (line : 3 column : 103)
ERROR:no viable alternative at input 'details' (line : 3 column : 109)
ERROR:missing '.' at 'UserA' (line : 3 column : 117)
ERROR:no viable alternative at input 'be' (line : 3 column : 128)
ERROR:no viable alternative at input 'unable' (line : 3 column : 131)
ERROR:missing '.' at 'impersonate' (line : 3 column : 141)
ERROR:no viable alternative at input 'them' (line : 3 column : 153)
ERROR:Couldn't resolve reference to Actor 'customer'. (line : 3 column : 21)
ERROR:Couldn't resolve reference to Actor 'email'. (line : 3 column : 35)
ERROR:Couldn't resolve reference to Actor 'first'. (line : 3 column : 66)
ERROR:Couldn't resolve reference to Actor 'views'. (line : 3 column : 97)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 91)
ERROR:Couldn't resolve reference to Actor 'their'. (line : 3 column : 103)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 128)
ERROR:Couldn't resolve reference to Actor 'impersonate'. (line : 3 column : 141)
```
### DSL-Rimay Score
DSL_Score: 72
Progressbar: ![72%](https://progress-bar.dev/72)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 96.0
Progressbar: ![96.0%](https://progress-bar.dev/96.0)
@@ -0,0 +1,126 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:27:14
* UniqueID: fb56e8b7-eb34-4df5-b1db-298b0453e692
* Gherkinscenario name: Gherkinscenario 1: Requesting the cash from a creditworthy account_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: that the account is creditworthy
When: UserA requests the cash
Then: ensure the account is debited
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: that the account is creditworthy
When: UserA requests the cash
Then: ensure the account is debited
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
ensure the account is debited should When UserA requests the cash
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'the' (line : 3 column : 8)
ERROR:missing '.' at 'account' (line : 3 column : 12)
ERROR:no viable alternative at input 'is' (line : 3 column : 20)
ERROR:no viable alternative at input 'debited' (line : 3 column : 23)
ERROR:no viable alternative at input 'should' (line : 3 column : 31)
ERROR:missing '.' at 'When' (line : 3 column : 38)
ERROR:mismatched input '<EOF>' expecting ',' (line : 3 column : 62)
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 12)
ERROR:Couldn't resolve reference to Actor 'debited'. (line : 3 column : 23)
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 62)
```
### DSL-Rimay Score
DSL_Score: 73
Progressbar: ![73%](https://progress-bar.dev/73)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 1
Incomplete system response: 1
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 70
Progressbar: ![70%](https://progress-bar.dev/70)
### Final Score
Final_score = 61.5
Progressbar: ![61.5%](https://progress-bar.dev/61.5)
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:26:19
* UniqueID: ff0caa2e-4982-466c-bdb9-df9daa946c5c
* Gherkinscenario name: Gherkinscenario 1: Requesting the cash from a creditworthy account
* LLM-temperature: 1.3
## Gherkin Input
```
Given: that the account is creditworthy
When: UserA requests the cash
Then: ensure the account is debited
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: that the account is creditworthy
When: UserA requests the cash
Then: ensure the account is debited
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While the account is creditworthy When UserA requests the cash UserA must ensure the account is debited.
```
### Researcher score
Researcher_score = 10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'When' (line : 3 column : 35)
ERROR:missing ',' at 'UserA' (line : 3 column : 64)
ERROR:no viable alternative at input 'ensure' (line : 3 column : 75)
ERROR:no viable alternative at input 'the' (line : 3 column : 82)
ERROR:missing '.' at 'account' (line : 3 column : 86)
ERROR:no viable alternative at input 'is' (line : 3 column : 94)
ERROR:no viable alternative at input 'debited' (line : 3 column : 97)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'account'. (line : 3 column : 11)
ERROR:Couldn't resolve reference to Quantifier 'creditworthy'. (line : 3 column : 22)
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 75)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 86)
```
### DSL-Rimay Score
DSL_Score: 82
Progressbar: ![82%](https://progress-bar.dev/82)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 88.5
Progressbar: ![88.5%](https://progress-bar.dev/88.5)
@@ -2,8 +2,8 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__13:30:02
* UniqueID: 6586ae09-fa25-44e1-8464-a5441e413989
* Datetime: 04_05_2024__14:29:31
* UniqueID: d56dc4e0-9b93-4f61-9e4e-3d478f3581ab
* Gherkinscenario name: Gherkinscenario 2: Money withdraw from bank account_Incorrect_
* LLM-temperature: 1.3
@@ -38,7 +38,7 @@ $ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirement
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
@@ -63,7 +63,7 @@ Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
The withdrawal should be complete without errors or warnings should While UserA bank account is in credit UserA When UserA attempts to withdraw an amount less than UserA card's limit
UserX should the withdrawal SystemA while UserA bank account is in credit Given the withdrawal completes without errors or warnings.
```
@@ -75,45 +75,32 @@ Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 16)
ERROR:missing '.' at 'be' (line : 3 column : 23)
ERROR:no viable alternative at input 'complete' (line : 3 column : 26)
ERROR:missing '.' at 'without' (line : 3 column : 35)
ERROR:no viable alternative at input 'errors' (line : 3 column : 43)
ERROR:missing '.' at 'warnings' (line : 3 column : 53)
ERROR:no viable alternative at input 'should' (line : 3 column : 63)
ERROR:missing '.' at 'While' (line : 3 column : 75)
ERROR:no viable alternative at input 'account' (line : 3 column : 92)
ERROR:no viable alternative at input 'is' (line : 3 column : 100)
ERROR:no viable alternative at input 'credit' (line : 3 column : 106)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 117)
ERROR:missing '.' at 'When' (line : 3 column : 126)
ERROR:no viable alternative at input 'to' (line : 3 column : 146)
ERROR:missing ',' at 'withdraw' (line : 3 column : 149)
ERROR:no viable alternative at input 'an' (line : 3 column : 158)
ERROR:missing '.' at 'amount' (line : 3 column : 161)
ERROR:no viable alternative at input 'less' (line : 3 column : 168)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 178)
ERROR:no viable alternative at input 'card' (line : 3 column : 184)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 188)
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 5)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 23)
ERROR:Couldn't resolve reference to Actor 'without'. (line : 3 column : 35)
ERROR:Couldn't resolve reference to Actor 'warnings'. (line : 3 column : 53)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 87)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 81)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 92)
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 106)
ERROR:Couldn't resolve reference to Actor 'attempts'. (line : 3 column : 137)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 131)
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 149)
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 161)
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'the' (line : 3 column : 14)
ERROR:no viable alternative at input 'SystemA' (line : 3 column : 29)
ERROR:missing '.' at 'while' (line : 3 column : 37)
ERROR:no viable alternative at input 'account' (line : 3 column : 54)
ERROR:no viable alternative at input 'is' (line : 3 column : 62)
ERROR:no viable alternative at input 'credit' (line : 3 column : 68)
ERROR:no viable alternative at input 'Given' (line : 3 column : 75)
ERROR:missing '.' at 'the' (line : 3 column : 81)
ERROR:no viable alternative at input 'completes' (line : 3 column : 96)
ERROR:missing '.' at 'without' (line : 3 column : 106)
ERROR:no viable alternative at input 'errors' (line : 3 column : 114)
ERROR:extraneous input 'warnings' expecting '.' (line : 3 column : 124)
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 18)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 49)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 43)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 54)
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 68)
ERROR:Couldn't resolve reference to Actor 'withdrawal'. (line : 3 column : 85)
ERROR:Couldn't resolve reference to Actor 'without'. (line : 3 column : 106)
```
### DSL-Rimay Score
DSL_Score: 51
DSL_Score: 74
Progressbar: ![51%](https://progress-bar.dev/51)
Progressbar: ![74%](https://progress-bar.dev/74)
@@ -128,21 +115,21 @@ Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Incomplete system response: 1
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
PASKA_Score: 80
Progressbar: ![95%](https://progress-bar.dev/95)
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 63.0
Final_score = 67.0
Progressbar: ![63.0%](https://progress-bar.dev/63.0)
Progressbar: ![67.0%](https://progress-bar.dev/67.0)
@@ -2,8 +2,8 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__13:29:05
* UniqueID: 8f138f36-7983-4012-a4be-07be403c98f2
* Datetime: 04_05_2024__14:28:09
* UniqueID: b3aba5d9-45d2-4fff-b984-bdca3aecfb01
* Gherkinscenario name: Gherkinscenario 2: Money withdraw from bank account
* LLM-temperature: 1.3
@@ -38,7 +38,7 @@ $ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirement
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
@@ -67,7 +67,7 @@ Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA bank account is in credit When UserA attempts to withdraw an amount less than UserA card's limit UserA must the withdrawal should be complete without errors or warnings
While UserA bank account is in credit When UserA attempts to withdraw an amount less than UserA card's limit UserA must the withdrawal should be complete without errors or warnings
```
@@ -82,22 +82,22 @@ Researcher_score = 10
ERROR:no viable alternative at input 'account' (line : 3 column : 18)
ERROR:no viable alternative at input 'is' (line : 3 column : 26)
ERROR:no viable alternative at input 'credit' (line : 3 column : 32)
ERROR:no viable alternative at input 'When' (line : 3 column : 41)
ERROR:missing '.' at 'UserA' (line : 3 column : 46)
ERROR:no viable alternative at input 'attempts' (line : 3 column : 52)
ERROR:missing '.' at 'withdraw' (line : 3 column : 64)
ERROR:no viable alternative at input 'an' (line : 3 column : 73)
ERROR:missing '.' at 'amount' (line : 3 column : 76)
ERROR:no viable alternative at input 'less' (line : 3 column : 83)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 93)
ERROR:no viable alternative at input 'card' (line : 3 column : 99)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 103)
ERROR:no viable alternative at input 'When' (line : 3 column : 39)
ERROR:missing '.' at 'UserA' (line : 3 column : 44)
ERROR:no viable alternative at input 'attempts' (line : 3 column : 50)
ERROR:missing '.' at 'withdraw' (line : 3 column : 62)
ERROR:no viable alternative at input 'an' (line : 3 column : 71)
ERROR:missing '.' at 'amount' (line : 3 column : 74)
ERROR:no viable alternative at input 'less' (line : 3 column : 81)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 91)
ERROR:no viable alternative at input 'card' (line : 3 column : 97)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 101)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'bank'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 18)
ERROR:Couldn't resolve reference to Actor 'credit'. (line : 3 column : 32)
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 64)
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 76)
ERROR:Couldn't resolve reference to Actor 'withdraw'. (line : 3 column : 62)
ERROR:Couldn't resolve reference to Actor 'amount'. (line : 3 column : 74)
```
### DSL-Rimay Score
@@ -0,0 +1,126 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:32:06
* UniqueID: 2ff9edd1-f8bb-4696-bdc7-7ba662800dbd
* Gherkinscenario name: Gherkinscenario 3: Requesting the cash from an overdrawn account_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: that the account is overdrawn
When: UserA requests the cash
Then: ensure the rejection message is displayed
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: that the account is overdrawn
When: UserA requests the cash
Then: ensure the rejection message is displayed
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
ensure the rejection message is displayed should While UserA requests the cash.
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'the' (line : 3 column : 8)
ERROR:missing '.' at 'rejection' (line : 3 column : 12)
ERROR:no viable alternative at input 'message' (line : 3 column : 22)
ERROR:missing '.' at 'should' (line : 3 column : 43)
ERROR:no viable alternative at input 'While' (line : 3 column : 50)
ERROR:missing '.' at 'UserA' (line : 3 column : 56)
ERROR:no viable alternative at input 'requests' (line : 3 column : 62)
ERROR:Couldn't resolve reference to Actor 'ensure'. (line : 3 column : 1)
ERROR:Couldn't resolve reference to Actor 'rejection'. (line : 3 column : 12)
ERROR:Couldn't resolve reference to Actor 'should'. (line : 3 column : 43)
ERROR:Couldn't resolve reference to Actor 'cash'. (line : 3 column : 75)
```
### DSL-Rimay Score
DSL_Score: 83
Progressbar: ![83%](https://progress-bar.dev/83)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 1
Incomplete system response: 1
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 70
Progressbar: ![70%](https://progress-bar.dev/70)
### Final Score
Final_score = 66.5
Progressbar: ![66.5%](https://progress-bar.dev/66.5)
@@ -0,0 +1,130 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:31:05
* UniqueID: 7ca06445-5795-4627-8459-9748e395bcb3
* Gherkinscenario name: Gherkinscenario 3: Requesting the cash from an overdrawn account
* LLM-temperature: 1.3
## Gherkin Input
```
Given: that the account is overdrawn
When: UserA requests the cash
Then: ensure the rejection message is displayed
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: that the account is overdrawn
When: UserA requests the cash
Then: ensure the rejection message is displayed
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While the account is overdrawn When UserA $MODAL_VERB ensure the rejection message is displayed
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'When' (line : 3 column : 32)
ERROR:no viable alternative at input '$' (line : 3 column : 43)
ERROR:missing ',' at 'MODAL_VERB' (line : 3 column : 44)
ERROR:no viable alternative at input 'ensure' (line : 3 column : 55)
ERROR:missing '.' at 'the' (line : 3 column : 62)
ERROR:no viable alternative at input 'message' (line : 3 column : 76)
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 87)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'account'. (line : 3 column : 11)
ERROR:Couldn't resolve reference to Quantifier 'overdrawn'. (line : 3 column : 22)
ERROR:Couldn't resolve reference to Actor 'MODAL_VERB'. (line : 3 column : 44)
ERROR:Couldn't resolve reference to Actor 'rejection'. (line : 3 column : 66)
```
### DSL-Rimay Score
DSL_Score: 73
Progressbar: ![73%](https://progress-bar.dev/73)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 64.0
Progressbar: ![64.0%](https://progress-bar.dev/64.0)
@@ -0,0 +1,131 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:33:56
* UniqueID: c987e803-7214-4ede-bc11-4863e9a39475
* Gherkinscenario name: Gherkinscenario 4: User presses button._Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA navigates to the login page
When: UserA clicks on button
Then: SystemA refreshes page
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA navigates to the login page
When: UserA clicks on button
Then: SystemA refreshes page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
SystemA refreshes page should When UserA navigates to the login page While UserA clicks on button
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'refreshes' (line : 3 column : 9)
ERROR:missing '.' at 'page' (line : 3 column : 19)
ERROR:no viable alternative at input 'should' (line : 3 column : 24)
ERROR:missing '.' at 'When' (line : 3 column : 31)
ERROR:no viable alternative at input 'to' (line : 3 column : 52)
ERROR:missing ',' at 'the' (line : 3 column : 55)
ERROR:no viable alternative at input 'page' (line : 3 column : 65)
ERROR:missing '.' at 'While' (line : 3 column : 70)
ERROR:no viable alternative at input 'clicks' (line : 3 column : 76)
ERROR:no viable alternative at input 'clicks' (line : 3 column : 82)
ERROR:no viable alternative at input 'button' (line : 3 column : 92)
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 19)
ERROR:Couldn't resolve reference to Actor 'navigates'. (line : 3 column : 42)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 36)
ERROR:Couldn't resolve reference to Actor 'login'. (line : 3 column : 59)
ERROR:Couldn't resolve reference to Actor 'button'. (line : 3 column : 92)
```
### DSL-Rimay Score
DSL_Score: 78
Progressbar: ![78%](https://progress-bar.dev/78)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 1
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 80
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 69.0
Progressbar: ![69.0%](https://progress-bar.dev/69.0)
@@ -0,0 +1,133 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:33:00
* UniqueID: d0a1cc5d-328f-4c1f-be7b-3842389aed2e
* Gherkinscenario name: Gherkinscenario 4: User presses button.
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA navigates to the login page
When: UserA clicks on button
Then: SystemA refreshes page
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA navigates to the login page
When: UserA clicks on button
Then: SystemA refreshes page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA navigates to the login page When UserA clicks on button SystemA must refresh page
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'to' (line : 3 column : 23)
ERROR:missing ',' at 'the' (line : 3 column : 26)
ERROR:no viable alternative at input 'page' (line : 3 column : 36)
ERROR:missing '.' at 'When' (line : 3 column : 41)
ERROR:missing RULE_STRING at 'SystemA' (line : 3 column : 69)
ERROR:mismatched input 'must' expecting ',' (line : 3 column : 77)
ERROR:missing '.' at 'refresh' (line : 3 column : 82)
ERROR:no viable alternative at input 'page' (line : 3 column : 90)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'navigates'. (line : 3 column : 13)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 7)
ERROR:Couldn't resolve reference to Actor 'login'. (line : 3 column : 30)
ERROR:Couldn't resolve reference to Quantifier 'button'. (line : 3 column : 62)
ERROR:Couldn't resolve reference to UIComponent 'SystemA'. (line : 3 column : 69)
ERROR:Couldn't resolve reference to Actor 'refresh'. (line : 3 column : 82)
```
### DSL-Rimay Score
DSL_Score: 73
Progressbar: ![73%](https://progress-bar.dev/73)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 89.0
Progressbar: ![89.0%](https://progress-bar.dev/89.0)
@@ -0,0 +1,154 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:36:50
* UniqueID: db1cf516-5278-4896-86b3-52920935f843
* Gherkinscenario name: Gherkinscenario 5: Getting verified after clicking the link in the verification message_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: a verification email has already been sent to "UserA"
When: UserA tries to verify his account using the link from this email
Then: UserA should be notified that the verification was successful
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "UserA"
When: UserA tries to verify his account using the link from this email
Then: UserA should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should While UserA tries to verify his account using the link from this email acting Notify interface, can verification emails to `$ACTOR $MODAL_VERB verifies account notify success.
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'While' (line : 3 column : 14)
ERROR:no viable alternative at input 'to' (line : 3 column : 32)
ERROR:missing ',' at 'verify' (line : 3 column : 35)
ERROR:no viable alternative at input 'his' (line : 3 column : 42)
ERROR:missing '.' at 'account' (line : 3 column : 46)
ERROR:no viable alternative at input 'using' (line : 3 column : 54)
ERROR:no viable alternative at input 'the' (line : 3 column : 60)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 64)
ERROR:missing '.' at 'this' (line : 3 column : 74)
ERROR:no viable alternative at input 'email' (line : 3 column : 79)
ERROR:missing '.' at 'acting' (line : 3 column : 85)
ERROR:no viable alternative at input 'Notify' (line : 3 column : 92)
ERROR:missing '.' at 'interface' (line : 3 column : 99)
ERROR:no viable alternative at input ',' (line : 3 column : 108)
ERROR:no viable alternative at input 'can' (line : 3 column : 110)
ERROR:no viable alternative at input 'verification' (line : 3 column : 114)
ERROR:missing '.' at 'emails' (line : 3 column : 127)
ERROR:no viable alternative at input 'to' (line : 3 column : 134)
ERROR:no viable alternative at input 'ACTOR' (line : 3 column : 139)
ERROR:no viable alternative at input '$' (line : 3 column : 145)
ERROR:no viable alternative at input 'MODAL_VERB' (line : 3 column : 146)
ERROR:no viable alternative at input 'verifies' (line : 3 column : 157)
ERROR:missing '.' at 'account' (line : 3 column : 166)
ERROR:no viable alternative at input 'notify' (line : 3 column : 174)
ERROR:mismatched input '.' expecting RULE_STRING (line : 3 column : 188)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'tries'. (line : 3 column : 26)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 20)
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 35)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 74)
ERROR:Couldn't resolve reference to Actor 'acting'. (line : 3 column : 85)
ERROR:Couldn't resolve reference to Actor 'interface'. (line : 3 column : 99)
ERROR:Couldn't resolve reference to Actor 'can'. (line : 3 column : 110)
ERROR:Couldn't resolve reference to Actor 'emails'. (line : 3 column : 127)
ERROR:Couldn't resolve reference to Actor 'ACTOR'. (line : 3 column : 139)
ERROR:Couldn't resolve reference to Actor 'MODAL_VERB'. (line : 3 column : 146)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 166)
ERROR:Couldn't resolve reference to Quantifier 'success'. (line : 3 column : 181)
```
### DSL-Rimay Score
DSL_Score: 48
Progressbar: ![48%](https://progress-bar.dev/48)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 64.0
Progressbar: ![64.0%](https://progress-bar.dev/64.0)
@@ -0,0 +1,153 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:35:23
* UniqueID: f5c1d46b-e6b5-40dd-b8f9-4be9d5eb74c1
* Gherkinscenario name: Gherkinscenario 5: Getting verified after clicking the link in the verification message
* LLM-temperature: 1.3
## Gherkin Input
```
Given: a verification email has already been sent to "UserA"
When: UserA tries to verify his account using the link from this email
Then: UserA should be notified that the verification was successful
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: a verification email has already been sent to "UserA"
When: UserA tries to verify his account using the link from this email
Then: UserA should be notified that the verification was successful
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While a verification email has already been sent to "UserA" When UserA tries to verify his account using the link from this email UserA must be notified that the verification was successful
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'email' (line : 3 column : 22)
ERROR:no viable alternative at input 'has' (line : 3 column : 28)
ERROR:no viable alternative at input 'already' (line : 3 column : 32)
ERROR:no viable alternative at input 'been' (line : 3 column : 40)
ERROR:missing '.' at 'sent' (line : 3 column : 45)
ERROR:no viable alternative at input 'to' (line : 3 column : 50)
ERROR:no viable alternative at input 'When' (line : 3 column : 61)
ERROR:no viable alternative at input 'to' (line : 3 column : 78)
ERROR:missing ',' at 'verify' (line : 3 column : 81)
ERROR:no viable alternative at input 'his' (line : 3 column : 88)
ERROR:missing '.' at 'account' (line : 3 column : 92)
ERROR:no viable alternative at input 'using' (line : 3 column : 100)
ERROR:no viable alternative at input 'the' (line : 3 column : 106)
ERROR:mismatched input 'link' expecting RULE_ID (line : 3 column : 110)
ERROR:missing '.' at 'this' (line : 3 column : 120)
ERROR:no viable alternative at input 'email' (line : 3 column : 125)
ERROR:missing '.' at 'UserA' (line : 3 column : 131)
ERROR:no viable alternative at input 'be' (line : 3 column : 142)
ERROR:no viable alternative at input 'notified' (line : 3 column : 145)
ERROR:missing '.' at 'the' (line : 3 column : 159)
ERROR:no viable alternative at input 'was' (line : 3 column : 176)
ERROR:missing '.' at 'successful' (line : 3 column : 180)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'verification'. (line : 3 column : 9)
ERROR:Couldn't resolve reference to Actor 'email'. (line : 3 column : 22)
ERROR:Couldn't resolve reference to Actor 'already'. (line : 3 column : 32)
ERROR:Couldn't resolve reference to Actor 'sent'. (line : 3 column : 45)
ERROR:Couldn't resolve reference to Actor 'tries'. (line : 3 column : 72)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 66)
ERROR:Couldn't resolve reference to Actor 'verify'. (line : 3 column : 81)
ERROR:Couldn't resolve reference to Actor 'account'. (line : 3 column : 92)
ERROR:Couldn't resolve reference to Actor 'this'. (line : 3 column : 120)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 142)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 163)
ERROR:Couldn't resolve reference to Actor 'successful'. (line : 3 column : 180)
```
### DSL-Rimay Score
DSL_Score: 60
Progressbar: ![60%](https://progress-bar.dev/60)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 82.5
Progressbar: ![82.5%](https://progress-bar.dev/82.5)
@@ -0,0 +1,140 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:39:03
* UniqueID: 8210419d-1fa6-4b5a-90b9-ecf3231ee025
* Gherkinscenario name: Gherkinscenario 6: The selected province stays after validation error_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA is editing the address of "Fletcher Ren"
When: UserA removes the street
Then: UserA should still be on the "Fletcher Ren" address edit page
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA is editing the address of "Fletcher Ren"
When: UserA removes the street
Then: UserA should still be on the "Fletcher Ren" address edit page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should still be on the "Fletcher Ren" address edit page should While UserA is editing the address of "Fletcher Ren" given the Gherkin Given UserA removes the street
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'still' (line : 3 column : 14)
ERROR:no viable alternative at input 'be' (line : 3 column : 20)
ERROR:missing '.' at 'the' (line : 3 column : 26)
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 30)
ERROR:no viable alternative at input 'edit' (line : 3 column : 53)
ERROR:no viable alternative at input 'should' (line : 3 column : 63)
ERROR:missing '.' at 'While' (line : 3 column : 70)
ERROR:no viable alternative at input 'the' (line : 3 column : 93)
ERROR:no viable alternative at input 'of' (line : 3 column : 105)
ERROR:no viable alternative at input 'given' (line : 3 column : 123)
ERROR:no viable alternative at input 'the' (line : 3 column : 129)
ERROR:missing '.' at 'Gherkin' (line : 3 column : 133)
ERROR:no viable alternative at input 'Given' (line : 3 column : 141)
ERROR:missing '.' at 'UserA' (line : 3 column : 147)
ERROR:no viable alternative at input 'removes' (line : 3 column : 153)
ERROR:mismatched input '<EOF>' expecting '.' (line : 3 column : 165)
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 45)
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 58)
ERROR:Couldn't resolve reference to Quantifier 'editing'. (line : 3 column : 85)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 97)
ERROR:Couldn't resolve reference to Actor 'given'. (line : 3 column : 123)
ERROR:Couldn't resolve reference to Actor 'Gherkin'. (line : 3 column : 133)
ERROR:Couldn't resolve reference to PropertyOrLabel 'street'. (line : 3 column : 165)
```
### DSL-Rimay Score
DSL_Score: 59
Progressbar: ![59%](https://progress-bar.dev/59)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 67.0
Progressbar: ![67.0%](https://progress-bar.dev/67.0)
@@ -0,0 +1,135 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:37:44
* UniqueID: 3f4f574e-7b38-457e-b68a-ec5140bdb4e9
* Gherkinscenario name: Gherkinscenario 6: The selected province stays after validation error
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA is editing the address of "Fletcher Ren"
When: UserA removes the street
Then: UserA should still be on the "Fletcher Ren" address edit page
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA is editing the address of "Fletcher Ren"
When: UserA removes the street
Then: UserA should still be on the "Fletcher Ren" address edit page
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While UserA is editing the address of "Fletcher Ren" When UserA removes the street UserA shall still be on the "Fletcher Ren" address edit page
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'the' (line : 3 column : 24)
ERROR:no viable alternative at input 'of' (line : 3 column : 36)
ERROR:no viable alternative at input 'When' (line : 3 column : 54)
ERROR:missing ',' at 'UserA' (line : 3 column : 84)
ERROR:no viable alternative at input 'still' (line : 3 column : 96)
ERROR:no viable alternative at input 'be' (line : 3 column : 102)
ERROR:missing '.' at 'the' (line : 3 column : 108)
ERROR:extraneous input '"Fletcher Ren"' expecting RULE_ID (line : 3 column : 112)
ERROR:no viable alternative at input 'edit' (line : 3 column : 135)
ERROR:no viable alternative at input '<EOF>' (line : 3 column : 140)
ERROR:Couldn't resolve reference to Quantifier 'editing'. (line : 3 column : 16)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 28)
ERROR:Couldn't resolve reference to PropertyOrLabel 'street'. (line : 3 column : 77)
ERROR:Couldn't resolve reference to Actor 'still'. (line : 3 column : 96)
ERROR:Couldn't resolve reference to Actor 'address'. (line : 3 column : 127)
ERROR:Couldn't resolve reference to Actor 'page'. (line : 3 column : 140)
```
### DSL-Rimay Score
DSL_Score: 68
Progressbar: ![68%](https://progress-bar.dev/68)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 86.5
Progressbar: ![86.5%](https://progress-bar.dev/86.5)
@@ -0,0 +1,127 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:41:03
* UniqueID: 7a49ec63-e37a-46be-9c97-73a97046d264
* Gherkinscenario name: Gherkinscenario 7: Not being able to add a product to the cart when it is out of stock_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: the product "T-shirt banana" is out of stock
When: UserA checks this product's details
Then: UserA should see that it is out of stock
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: the product "T-shirt banana" is out of stock
When: UserA checks this product's details
Then: UserA should see that it is out of stock
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should see that "T-shirt banana" is out of stock should UserA check this product's details
```
### Researcher score
Researcher_score = -5
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'see' (line : 3 column : 14)
ERROR:no viable alternative at input 'that' (line : 3 column : 18)
ERROR:no viable alternative at input 'stock' (line : 3 column : 50)
ERROR:no viable alternative at input 'should' (line : 3 column : 56)
ERROR:missing '.' at 'UserA' (line : 3 column : 63)
ERROR:no viable alternative at input 'check' (line : 3 column : 69)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 87)
ERROR:Couldn't resolve reference to Actor 'see'. (line : 3 column : 14)
ERROR:Couldn't resolve reference to Actor 'stock'. (line : 3 column : 50)
ERROR:Couldn't resolve reference to Quantifier 'this'. (line : 3 column : 75)
ERROR:Couldn't resolve reference to ClassOrLabel 'product'. (line : 3 column : 80)
```
### DSL-Rimay Score
DSL_Score: 72
Progressbar: ![72%](https://progress-bar.dev/72)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 0
```
### Paska Score
PASKA_Score: 100
Progressbar: ![100%](https://progress-bar.dev/100)
### Final Score
Final_score = 81.0
Progressbar: ![81.0%](https://progress-bar.dev/81.0)
@@ -0,0 +1,129 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:39:56
* UniqueID: a9bd72dc-e8b8-4ef7-898c-43d94a36b059
* Gherkinscenario name: Gherkinscenario 7: Not being able to add a product to the cart when it is out of stock
* LLM-temperature: 1.3
## Gherkin Input
```
Given: the product "T-shirt banana" is out of stock
When: UserA checks this product's details
Then: UserA should see that it is out of stock
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: the product "T-shirt banana" is out of stock
When: UserA checks this product's details
Then: UserA should see that it is out of stock
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While the product "T-shirt banana" is out of stock When UserA checks this product's details UserA must see that it is out of stock
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input '"T-shirt banana"' (line : 3 column : 19)
ERROR:missing ',' at 'stock' (line : 3 column : 46)
ERROR:no viable alternative at input 'When' (line : 3 column : 52)
ERROR:missing '.' at 'UserA' (line : 3 column : 57)
ERROR:no viable alternative at input 'checks' (line : 3 column : 63)
ERROR:mismatched character '<EOF>' expecting ''' (line : 3 column : 82)
ERROR:Couldn't resolve reference to ActorOrClassOrProperty 'product'. (line : 3 column : 11)
ERROR:Couldn't resolve reference to Actor 'stock'. (line : 3 column : 46)
ERROR:Couldn't resolve reference to Quantifier 'this'. (line : 3 column : 70)
ERROR:Couldn't resolve reference to ClassOrLabel 'product'. (line : 3 column : 75)
```
### DSL-Rimay Score
DSL_Score: 74
Progressbar: ![74%](https://progress-bar.dev/74)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 1
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 75
Progressbar: ![75%](https://progress-bar.dev/75)
### Final Score
Final_score = 89.5
Progressbar: ![89.5%](https://progress-bar.dev/89.5)
@@ -0,0 +1,145 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:42:52
* UniqueID: e6644809-f5a3-4705-8cdd-4206e476df36
* Gherkinscenario name: Gherkinscenario 8: Disabling country_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: UserA should be notified that it has been successfully edited
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: UserA should be notified that it has been successfully edited
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
UserA should While assert that UserA $modals country has been edited$gal once Triggered $systemAU textured SystemAY United States$galribelowhinewherit
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'should' (line : 3 column : 7)
ERROR:missing '.' at 'While' (line : 3 column : 14)
ERROR:no viable alternative at input 'that' (line : 3 column : 20)
ERROR:no viable alternative at input 'that' (line : 3 column : 27)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 32)
ERROR:no viable alternative at input '$' (line : 3 column : 38)
ERROR:no viable alternative at input 'modals' (line : 3 column : 39)
ERROR:no viable alternative at input 'country' (line : 3 column : 46)
ERROR:missing '.' at 'been' (line : 3 column : 58)
ERROR:no viable alternative at input 'edited' (line : 3 column : 63)
ERROR:missing '.' at 'gal' (line : 3 column : 70)
ERROR:no viable alternative at input 'once' (line : 3 column : 74)
ERROR:missing '.' at 'Triggered' (line : 3 column : 79)
ERROR:no viable alternative at input '$' (line : 3 column : 89)
ERROR:no viable alternative at input 'systemAU' (line : 3 column : 90)
ERROR:no viable alternative at input 'textured' (line : 3 column : 99)
ERROR:missing '.' at 'SystemAY' (line : 3 column : 108)
ERROR:no viable alternative at input 'United' (line : 3 column : 117)
ERROR:missing '.' at 'States' (line : 3 column : 124)
ERROR:no viable alternative at input '$' (line : 3 column : 130)
ERROR:no viable alternative at input 'galribelowhinewherit' (line : 3 column : 131)
ERROR:Couldn't resolve reference to Actor 'assert'. (line : 3 column : 20)
ERROR:Couldn't resolve reference to Actor 'modals'. (line : 3 column : 39)
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 58)
ERROR:Couldn't resolve reference to Actor 'gal'. (line : 3 column : 70)
ERROR:Couldn't resolve reference to Actor 'Triggered'. (line : 3 column : 79)
ERROR:Couldn't resolve reference to Actor 'systemAU'. (line : 3 column : 90)
ERROR:Couldn't resolve reference to Actor 'SystemAY'. (line : 3 column : 108)
ERROR:Couldn't resolve reference to Actor 'States'. (line : 3 column : 124)
ERROR:Couldn't resolve reference to Actor 'galribelowhinewherit'. (line : 3 column : 131)
```
### DSL-Rimay Score
DSL_Score: 64
Progressbar: ![64%](https://progress-bar.dev/64)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,1
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 0
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 80
Progressbar: ![80%](https://progress-bar.dev/80)
### Final Score
Final_score = 62.0
Progressbar: ![62.0%](https://progress-bar.dev/62.0)
@@ -0,0 +1,140 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:41:57
* UniqueID: cd538520-59c9-44be-9245-8363af6445cb
* Gherkinscenario name: Gherkinscenario 8: Disabling country
* LLM-temperature: 1.3
## Gherkin Input
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: UserA should be notified that it has been successfully edited
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: SystemA has country "United States"
When: UserA wants to edit this country
Then: UserA should be notified that it has been successfully edited
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
While SystemA has country "United States" When UserA wants to edit this country UserA shall be notified that it has been successfully edited
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'to' (line : 3 column : 60)
ERROR:missing ',' at 'edit' (line : 3 column : 63)
ERROR:no viable alternative at input 'this' (line : 3 column : 68)
ERROR:missing '.' at 'country' (line : 3 column : 73)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 81)
ERROR:missing '.' at 'be' (line : 3 column : 93)
ERROR:no viable alternative at input 'notified' (line : 3 column : 96)
ERROR:missing '.' at 'it' (line : 3 column : 110)
ERROR:no viable alternative at input 'has' (line : 3 column : 113)
ERROR:no viable alternative at input 'been' (line : 3 column : 117)
ERROR:no viable alternative at input 'successfully' (line : 3 column : 122)
ERROR:missing '.' at 'edited' (line : 3 column : 135)
ERROR:Couldn't resolve reference to Quantifier 'country'. (line : 3 column : 19)
ERROR:Couldn't resolve reference to Actor 'wants'. (line : 3 column : 54)
ERROR:Couldn't resolve reference to Quantifier 'UserA'. (line : 3 column : 48)
ERROR:Couldn't resolve reference to Actor 'edit'. (line : 3 column : 63)
ERROR:Couldn't resolve reference to Actor 'country'. (line : 3 column : 73)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 93)
ERROR:Couldn't resolve reference to Actor 'it'. (line : 3 column : 110)
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 117)
ERROR:Couldn't resolve reference to Actor 'edited'. (line : 3 column : 135)
```
### DSL-Rimay Score
DSL_Score: 73
Progressbar: ![73%](https://progress-bar.dev/73)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 99.0
Progressbar: ![99.0%](https://progress-bar.dev/99.0)
@@ -0,0 +1,116 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:45:13
* UniqueID: a6c4e910-ed52-4352-a840-be0ead2b31f0
* Gherkinscenario name: Gherkinscenario 9: Resending the verification email as a logged in user_Incorrect_
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA is logged in as "valkyrie@cain.com"
When: UserA resends the verification email
Then: UserA should be notified that the verification email has been sent
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: should or can
$TRIGGER is for example a trigger action, nothing what a system does.
$WHILE_STRUCTUREis the word 'When' followed by the Gherkin When condition
$WHEN_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $SYSTEM_RESPONSE $MODAL_VERB $WHILE_STRUCTURE $ACTOR $WHEN_STRUCTURE
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA is logged in as "valkyrie@cain.com"
When: UserA resends the verification email
Then: UserA should be notified that the verification email has been sent
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
=UserA should be notified that the verification email has been sent should UserA While UserA resends the verification email
```
### Researcher score
Researcher_score = -10
## DSL-Rimay Verification
```
ERROR:missing EOF at '=' (line : 3 column : 1)
```
### DSL-Rimay Score
DSL_Score: 83
Progressbar: ![83%](https://progress-bar.dev/83)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 1
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 2
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 79.0
Progressbar: ![79.0%](https://progress-bar.dev/79.0)
@@ -0,0 +1,140 @@
# LLM Experiment Information
* Prefix: Few-shot-learning
* Datetime: 04_05_2024__14:43:43
* UniqueID: 83374f34-15f4-4e37-a986-c0965cb51de1
* Gherkinscenario name: Gherkinscenario 9: Resending the verification email as a logged in user
* LLM-temperature: 1.3
## Gherkin Input
```
Given: UserA is logged in as "valkyrie@cain.com"
When: UserA resends the verification email
Then: UserA should be notified that the verification email has been sent
```
## Input prompt, technique: Few-Shot-Learning
### Context
We are translating Gherkin acceptance criteria into Rimay CNL.
For this translation use the following information:
$ACTOR is for example a thing or a person.
$MODAL_VERB is for the following word: must or shall.
$TRIGGER is for example a trigger action, something what a system does.
$WHILE_STRUCTURE is the word 'While' followed by the Gherkin Given condition
$WHEN_STRUCTURE is the word 'When' followed by the Gherkin When condition
$SYSTEM_RESPONSE is a combination of the Actors and the Gherkin Then condition.
$ACTOR, $MODAL_VERB and a $SYSTEM_RESPONSE is mandatory in all Rimay requirements.
Replace the variables with the right text according the input acceptance criteria.
In the following order:
Rimay CNL definition: $WHILE_STRUCTURE $WHEN_STRUCTURE $ACTOR $MODAL_VERB $SYSTEM_RESPONSE
The following example is a valid Rimay CNL:
When SystemB receives an "email alert" from SystemA, SystemB must send an Instruction to SystemC.
With the usage of the following extra information:
Actors: actor SystemA, actor SystemB, actor SystemC, actor UserX, actor UserA, actor UserC
### Input Data
Use the following Gherkin acceptance criteria input:
```
Given: UserA is logged in as "valkyrie@cain.com"
When: UserA resends the verification email
Then: UserA should be notified that the verification email has been sent
```
### Question:
Can you translate the input above into the specified language Rimay CNL?
### Output indicator
Output only the translated Rimay CNL text and nothing else!
## Rimay Output
```
When UserA is logged in as "valkyrie@cain.com", UserA must resend the verification email. UserA should be notified that the verification email has been sent.
```
### Researcher score
Researcher_score = 15
## DSL-Rimay Verification
```
ERROR:no viable alternative at input 'is' (line : 3 column : 12)
ERROR:missing ',' at 'logged' (line : 3 column : 15)
ERROR:no viable alternative at input 'in' (line : 3 column : 22)
ERROR:no viable alternative at input 'UserA' (line : 3 column : 49)
ERROR:no viable alternative at input 'resend' (line : 3 column : 60)
ERROR:no viable alternative at input 'the' (line : 3 column : 67)
ERROR:missing '.' at 'verification' (line : 3 column : 71)
ERROR:no viable alternative at input 'email' (line : 3 column : 84)
ERROR:no viable alternative at input 'should' (line : 3 column : 97)
ERROR:missing '.' at 'be' (line : 3 column : 104)
ERROR:no viable alternative at input 'notified' (line : 3 column : 107)
ERROR:missing '.' at 'the' (line : 3 column : 121)
ERROR:no viable alternative at input 'email' (line : 3 column : 138)
ERROR:missing '.' at 'been' (line : 3 column : 148)
ERROR:no viable alternative at input 'sent' (line : 3 column : 153)
ERROR:Couldn't resolve reference to Actor 'logged'. (line : 3 column : 15)
ERROR:Couldn't resolve reference to Actor 'resend'. (line : 3 column : 60)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 71)
ERROR:Couldn't resolve reference to Actor 'be'. (line : 3 column : 104)
ERROR:Couldn't resolve reference to Actor 'verification'. (line : 3 column : 125)
ERROR:Couldn't resolve reference to Actor 'been'. (line : 3 column : 148)
```
### DSL-Rimay Score
DSL_Score: 73
Progressbar: ![73%](https://progress-bar.dev/73)
## Rimay Paska Verification
```
File generated_rimay.csv.csv
Smells found in this document:
Non-atomic requirement: 0
Incomplete requirement: 0
Incorrect order requirement: 0
Coordination ambiguity: 0
Not requirement: ,0
Incomplete condition: 0
Incomplete system response: 0
Passive voice: 1
Not precise verb: 0
Number Req with smells: 1
```
### Paska Score
PASKA_Score: 95
Progressbar: ![95%](https://progress-bar.dev/95)
### Final Score
Final_score = 99.0
Progressbar: ![99.0%](https://progress-bar.dev/99.0)
+2 -2
View File
@@ -32,7 +32,7 @@ def convert_single_gherkin_to_rimay(scenario_name: str, input_text:str, translat
```
"""
current_logger.append_result(pre_content)
response = ask_different_prompts(input_text.strip(), translation_type, current_logger, incorrect, temperature)
response = ask_different_prompts(input_text.strip(), translation_type, current_logger, temperature, incorrect)
researcher_score(input_text, response, current_logger)
rimay_check(response, current_logger)
@@ -141,7 +141,7 @@ def results(path):
],
}).classes('w-full h-200')
}).classes('w-full h-400')
def update():
chart.options['series'][0]['data'][0] = random()