Add coinflip task

tasks/cot/coinflip.yaml

@@ -0,0 +1,14 @@
+name: 'coinflip'
+type: 'gen'
+module: "tasks.cot.task.ChainOfThoughtTask"
+path: 'symbolic'
+file-pattern:
+  test: "coinflip.jsonl"
+sampling_strategy: "BaseStrategy"
+prompt_path: "tasks/cot/coinflip_prompt.txt"
+deterministic: true
+max_gen_length: 64
+use_task_mask: true
+save_prediction: true
+chain_of_thought: true
+micro_batch_size: 4

tasks/cot/coinflip_prompt.txt

@@ -0,0 +1,16 @@
+Q: A coin is heads up. Ka flips the coin. Sherrie flips the coin. Is the coin still heads up?
+A: The coin was flipped by Ka and Sherrie. So the coin was flipped 2 times, which is an even number. The coin started heads up, so after an even number of flips, it will still be heads up. So the answer is yes.
+Q: A coin is heads up. Jamey flips the coin. Teressa flips the coin. Is the coin still heads up?
+A: The coin was flipped by Jamey and Teressa. So the coin was flipped 2 times, which is an even number. The coin started heads up, so after an even number of flips, it will still be heads up. So the answer is yes.
+Q: A coin is heads up. Maybelle flips the coin. Shalonda does not flip the coin. Is the coin still heads up?
+A: The coin was flipped by Maybelle. So the coin was flipped 1 time, which is an odd number. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no.
+Q: A coin is heads up. Millicent does not flip the coin. Conception flips the coin. Is the coin still heads up?
+A: The coin was flipped by Conception. So the coin was flipped 1 time, which is an odd number. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no.
+Q: A coin is heads up. Sal flips the coin. Raymond does not flip the coin. Is the coin still heads up?
+A: The coin was flipped by Sal. So the coin was flipped 1 time, which is an odd number. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no.
+Q: A coin is heads up. Conception flips the coin. Kristian does not flip the coin. Is the coin still heads up?
+A: The coin was flipped by Conception. So the coin was flipped 1 time, which is an odd number. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no.
+Q: A coin is heads up. Inga does not flip the coin. Elanor does not flip the coin. Is the coin still heads up?
+A: The coin was flipped by no one. So the coin was flipped O times. The coin started heads up, and it was not flipped, so it is still heads up. So the answer is yes.
+Q: A coin is heads up. Ryan flips the coin. Shaunda flips the coin. Is the coin still heads up?
+A: The coin was flipped by Ryan and Shaunda. So the coin was flipped 2 times, which is an even number. The coin started heads up, so after an even number of flips, it will still be heads up. So the answer is yes.

tasks/cot/lastletter.yaml

@@ -7,6 +7,7 @@ file-pattern:
 sampling_strategy: "BaseStrategy"
 prompt_path: "tasks/cot/lastletter_prompt.txt"
 deterministic: true
+unidirectional: true
 max_gen_length: 64
 use_task_mask: true
 save_prediction: true

tasks/cot/sports.yaml

@@ -7,8 +7,10 @@ file-pattern:
 sampling_strategy: "BaseStrategy"
 prompt_path: "tasks/cot/sports_prompt.txt"
 deterministic: true
+unidirectional: true
 max_gen_length: 128
 use_task_mask: true
 save_prediction: true
 chain_of_thought: true
+prompt_type: 'number'
 micro_batch_size: 4

tasks/cot/task.py

@@ -63,7 +63,7 @@ def extract_answer(prediction, task_name, chain_of_thought=True):
             answer = match.group(0)
         else:
             answer = ""
-    elif task_name.startswith("sports"):
+    elif task_name.startswith("sports") or task_name.startswith("coinflip"):
         prediction = prediction.lower()
         if chain_of_thought:
             pattern = r'(?<=the answer is )(yes|no)'
@@ -176,6 +176,8 @@ class ChainOfThoughtTask(GenerationTask):
             return SportsDataset(os.path.join(self.config.path, relative_path), self.config)
         elif self.config.name.startswith("lastletter"):
             return LastLetterDataset(os.path.join(self.config.path, relative_path), self.config)
+        elif self.config.name.startswith("coinflip"):
+            return ChainOfThoughtDataset(os.path.join(self.config.path, relative_path), self.config)
         else:
             raise NotImplementedError